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Preface 


The  purpose  of  this  study  was  to  determine  the 
extent  to  which  AFCC  Operational  Test  and  Evaluation 
reports  were  used  to  aid  in  the  program  decision-making 
process  for  communication  systems.  Much  of  the  research 
that  has  been  accomplished  on  Operational  Test  and  Evalua¬ 
tion  has  primarily  focused  on  its  effect  on  major  programs. 
The  focus  of  this  research  has  been  on  non-major  programs 
at  the  major  command  (MAJCOM)  level. 

Review  of  the  available  literature  and  regulatory 
guidance  combined  with  interviews  of  program  managers  and 
test  directors  revealed  that  operational  testing  is  very 
different  for  minor  programs.  Quite  often  the  purpose  is 
more  to  identify  deficiencies  than  to  have  a  significant 
input  into  the  program  decision-making  process.  However, 
this  is  not  to  say  that  this  practice  is  wrong.  Many  of 
the  minor  programs  use  commercial  off-the-shelf  equipment 
modified  for  military  use  or  equipment  already  in  the 
military  inventory  but  reconfigured  to  satisfy  changing 
requirements.  For  these  types  of  systems  the  greatest 
value  of  operational  testing  is  to  identify  problems  with 
interoperability  since  many  of  the  individual  equipment 
items  have  already  been  extensively  tested  separately. 
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There  is  a  definite  need  for  operational  testing  at  every 
level  from  the  major  program  to  the  non-major  program. 
Without  it  the  Air  Force  would  not  be  assured  of  procuring 
the  best  systems  possible.  It  is  hoped  that  this  paper 
will  serve  to  enhance  the  understanding  of  how  operational 
test  and  evaluation  is  conducted  for  non-major  programs 
at  the  MAJCOM  level  and  specifically  within  Air  Force  Com¬ 
munications  Command. 

In  conducting  this  research  I  have  become  indebted 
to  a  great  number  of  people.  I  wish  to  thank  my  thesis 
advisor,  James  D.  Meadows,  and  Lt.  Col.  John  Diamond  for 
their  patience  and  guidance  in  this  effort.  I  would  also 
like  to  acknowledge  the  invaluable  assistance  of  the  mem¬ 
bers  of  the  1815th  Test  and  Evaluation  Squadron.  They  are 
true  professionals  and  I  have  thoroughly  enjoyed  working 
with  them  throughout  this  research  effort.  Special  thanks 
is  also  owed  to  Phyllis  Reynolds  for  the  care  with  which 
she  typed  these  pages.  Finally,  I  would  like  to  thank  my 
family  and  friends  that  continually  supported  me  with  their 
prayers  and  encouragement. 


Lorraine  Y.  Roemish 
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Abstract 


The  use  of  operational  test  and  evaluation  (OT&E) 
results  in  the  acquisition  decision-making  process  has  been 
increasingly  emphasized  in  the  past  years  and  has  gained 
even  more  prominence  with  the  establishment  of  the  Opera¬ 
tional  Test  and  Evaluation  Office  at  the  Department  of 
Defense.  Special  emphasis  has  been  placed  on  the  use  of 
OT&E  in  the  production  decision.  Initial  Operational  Test 
and  Evaluation  (I.OT&E)  and  Qualification  Operational  Test 
and  Evaluation  (QOT&E)  are  the  types  of  test  and  evalua¬ 
tion  that  are  normally  conducted  on  systems  prior  to  the 
first  major  production  decision  in  the  acquisition  process. 
While  IOT&E  usually  evaluates  new  systems  and  QOT&E  usually 
evaluates  off-the-shelf  systems,  they  both  provide  program 
decision  makers  with  valid  estimates  of  operational  effec¬ 
tiveness  and  operational  suitability  upon  which  they  can 
base  their  decisions.  This  research  has  looked  at  how 
ICT&E  and  QOT&E  reports  written  by  the  1815th  Test  and 
Evaluation  Squadron  (AFCC)  are  used  in  the  acquisition  of 
non-major  communication  systems.  Tne  results  of  this 
research  reveal  that  the  OT&E  conducted  by  the  1815th  is 
usually  QOT&E  performed  on  off-the-shelf  systems.  Since 
these  systems  involve  very  little  technical  risk  the  OT&E 


results  are  not  usually  the  primary  factor  in  production 
decisions.  OT&E  results  are  most  useful  in  pointing  out 
deficiencies  requiring  correction  prior  to  fielding  such 
systems . 


INITIAL  AND  QUALIFICATION  OPERATIONAL  TEST 


AND  EVALUATION  IN  THE  ACQUISITION  OF 
NON-MAJOR  COMMUNICATION  SYSTEMS 


I.  Introduction 


Initial  Operational  Test  and  Evaluation  (IOT&E)  is 
an  important  phase  of  test  and  evaluation  performed  during 
the  Air  Force  acquisition  process.  IOT&E  is  conducted 
prior  to  the  first  major  production  decision  to  provide 
program  decision  makers  with  valid  estimates  of  the  opera¬ 
tional  effectiveness  and  operational  suitability  of  a  sys¬ 
tem.  It  is  the  type  of  test  and  evaluation  of  foremost 
concern  to  the  major  command  that  will  eventually  have  the 
responsibility  for  operating  and  supporting  a  new  system. 
IOT&E  is  usually  the  first  opportunity  that  the  using 
command  has  to  evaluate  how  well  the  system  fulfills  the 
operational  need  it  was  designed  to  satisfy.  The  results 
of  the  IOT&E  effort  can  have  a  significant  impact  on  the 
acquisition  and  life  cycle  costs  for  a  system,  especially 
if  major  deficiencies  are  found  that  require  modification 
to  the  system  design.  Since  it  is  usually  much  easier  and 
much  less  expensive  to  correct  a  problem  the  earlier  it  is 
detected,  the  efficient  and  effective  performance  of  IOT&E 
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is  of  concern  to  the  Air  Force  and  to  each  major  command 
involved  in  the  acquisition  of  new  systems. 


Background 

The  primary  objective  of  this  research  effort  is 
to  determine  the  extent  to  which  IOT&E  conducted  by  the 
Air  Force  Communications  Command  (AFCC)  is  used  in  making 
production  decisions  in  the  acquisition  of  communication 
systems.  An  in-house  study  initiated  in  1983  by  AFCC  of 
the  overall  management  of  operational  test  and  evaluation 
(OT&E)  indicated  that  there  were  several  areas  that 
required  improvement  in  regard  to  the  conduct  of  OT&E  (16) . 
The  use  of  test  reports  in  the  acquisition  decision-making 
process  was  one  such  area. 

The  study  was  initiated  to  answer  questions  and 
resolve  problems  perceived  to  be  present  in  the  way  that 
test  and  evaluation  was  performed  by  AFCC.  Originally 
intended  to  be  conducted  by  an  independent  research  firm  in 
July  1983,  the  study  was  instead  tasked  to  the  1815th  Test 
and  Evaluation  Squadron  (TES)  in  August  1983  after  funding 
for  the  project  was  disapproved  (17) .  It  is  important  to 
note  that  the  1815th  TES  is  the  primary  organization  within 
AFCC  responsible  for  the  conduct  of  OT&E.  While  the 
results  of  the  study  cannot  be  considered  totally  without 
bias,  the  findings  indicated  an  earnest  effort  on  the  part 
of  the  study  team  to  provide  a  report  as  impartial  as 
possible . 


The  team  assigned  to  undertake  this  evaluation 
task  was  given  only  five  weeks  to  complete  the  study. 
Because  of  the  extent  of  the  tasking,  this  was  not  an  ade¬ 
quate  length  of  time  in  which  to  do  a  complete  and  thorough 
job.  Even  so,  many  of  their  findings  appear  valid  and 
warrant  further  investigation.  Major  problems  that  the 
team  identified  included  several  deficiencies  in  current 
regulations  applicable  to  operational  test  and  evaluation, 
inconsistencies  in  the  approval  policy  for  test  reports, 
inadequate  follow-up  on  test  recommendations,  and  a  lack 
of  training  for  OT&E  personnel  (16) .  The  specific  find¬ 
ings  that  generated  this  current  research  effort  were  the 
result  of  interviews  performed  by  the  research  team  of  the 
various  offices  tasked  with  either  generating  or  reviewing 
the  test  reports.  A  common  complaint  seemed  to  be  that, 
in  many  cases,  the  OT&E  results  and  recommendations  did 
not  appear  to  have  been  a  significant  factor  in  the  produc¬ 
tion  decisions.  The  implication  is  that  potential  cost 
savings  may  not  be  realized  if  the  recommendations  are 
ignored  when  making  the  production  decision.  Determining 
the  source  for  this  perception,  whether  or  not  it  is  valid, 
and  some  possible  solutions  comprise  the  central  thrust  of 


this  research. 


Definitions 


Operational  test  and  evaluation  (OT&E)  is  one  of 
two  general  kinds  of  test  and  evaluation  of  importance  in 
the  system  acquisition  process.  The  other  is  development 
test  and  evaluation  (DT&E) .  Both  of  these  are  operation¬ 
ally  defined  in  Air  Force  Regulation  (AFR)  80-14  and  Air 
Force  Regulation  (AFR) 55-43.  In  accordance  with  these 
documents,  DT&E  is  concerned  with  demonstrating  that  the 
system  engineering  design  and  development  is  complete, 
that  design  risks  have  been  minimized,  and  that  the  system 
will  perform  as  required  and  specified  (10:1;  11:2).  DT&E 
is  primarily  the  responsibility  of  the  implementing  command 
(usually  Air  Force  Systems  Command  or  Air  Force  Logistics 
Command)  and  the  contractor  providing  the  system. 

OT&E,  on  the  other  hand,  is  concerned  with  system 
performance  under  conditions  as  realistic  as  possible  in 
order  to  estimate  a  system's  operational  effectiveness  and 
operational  suitability,  to  identify  any  operational 
deficiencies,  and  to  identify  the  need  for  any  modifica¬ 
tions  (11:2).  In  order  to  understand  what  is  meant  by 
operational  effectiveness  and  operational  suitability, 

DODD  5000.1  offers  the  following  definitions: 

1.  Operational  Effectiveness .  The  overall  degree  of 
mission  accomplishment  of  a  system  used  by  representa¬ 
tive  personnel  in  the  context  of  the  organization, 
doctrine,  tactics,  threat  (including  countermeasures 
and  nuclear  threats)  and  environment  in  the  planned 
operational  employment  of  the  system. 


2.  Operational  Suitability .  The  degree  to  which  a 
system  can  be  placed  satisfactorily  in  field  use,  with 
consideration  being  given  to  availability,  compati¬ 
bility,  transportability,  interoperability,  reliability, 
wartime  usage  rates,  maintainability,  safety,  human 
factors,  manpower  supportability ,  logistic  support- 
ability,  and  training  requirements.  (3:3) 

These  definitions  show  the  wide  variety  of  issues  that 
OT&E  is  concerned  with.  In  essence,  operational  effective¬ 
ness  is  how  well  a  system  performs  its  intended  mission  in 
its  intended  environment,  while  operational  suitability  is 
how  well  the  system  is  suited  for  operation  and  maintenance 
by  military  personnel  in  the  field  (l:A-3).  Depending  on 
the  size  of  the  program,  OT&E  is  managed  and  conducted 
by  either  the  Air  Force  Operational  Test  and  Evaluation 
Center  (AFOTEC)  or  the  command  which  will  use  and  maintain 
the  system  when  it  is  deployed. 

Both  OT&E  and  DT&E  are  essential  and  complementary 
elements  of  the  systems  acquisition  process  and  occur 
throughout  the  system's  life  cycle.  However,  it  is  during 
OT&E  that  the  users  can  have  the  greatest  impact  on  the 
final  configuration  of  the  system  they  will  eventually  be 
tasked  to  operate. 

There  are  three  types  of  OT&E:  initial  OT&E 
( IOT&E) ,  follow-on  OT&E  (FOT&E) ,  and  qualification  OT&E 
(QOT&E) .  IOT&E  is  conducted  before  the  first  major  produc¬ 
tion  decision  using  a  prototype  or  a  preproduction  article. 
FOT&E  is  operational  testing  usually  conducted  after  the 
first  major  production  decision  or  after  the  first 


production  article  has  been  accepted.  QOT&E  is  performed 
instead  of  IOT&E  on  programs  where  there  is  no  funding  for 
research,  development,  test  and  evaluation  (RDT&E)  (11: 
2-3).  As  mentioned  earlier,  this  research  effort  is  pri¬ 
marily  concerned  with  IOT&E.  However,  since  QOT&E  is  per¬ 
formed  before  the  first  major  production  decision  and 
essentially  performs  the  same  type  of  testing,  but  on  com¬ 
mercial  off-the-shelf  equipment,  it  will  also  be  reviewed 
in  this  paper. 

Research  Questions 

The  major  research  question  for  this  thesis  con¬ 
cerns  the  usefulness  of  IOT&E  results  obtained  by  AFCC  to 
the  system  acquisition  process.  In  essence,  does  IOT&E 
serve  the  purpose  intended  according  to  current  regula¬ 
tions?  The  questions  chosen  to  pursue  this  line  of  thought 
follow. 

1.  Are  the  evaluation  reports  sufficient  in  terms 
of  practical  recommendations  to  provide  valid  input  into 
production  decisions?  If  not,  is  this  caused  by: 

a.  Insufficiently  trained  test  personnel, 

b.  Lack  of  inadequate  manpower, 

c.  Insufficient  guidelines  or  regulations  that 
should  be  providing  more  in-depth  information  on  testing 
procedures , 
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d.  Report  being  accomplished/prepared  too 
late  in  the  acquisition  process, 

e.  Inadequate  time, 

f.  A  combination  of  the  above, 

g.  Other  factors. 

2.  If  the  test  reports  are  deemed  sufficient,  are 
they  being  used  effectively  by  program  decision  makers? 

a.  What  is  the  perceived  value  of  the  test 
reports  to  the  overall  program  manager  (PM) ? 

b.  Are  the  test  results  being  considered  by 
the  PM  when  making  production  decisions  or  in  formulating 
recommendations  on  production  decisions  that  are  to  be  made 
by  higher  level  decision  makers?  In  what  way? 

c.  If  not  used  in  the  production  decision, 
why  not?  What  criteria  are  used  in  making  production 
decisions? 

d.  What  are  some  recommendations  for  improving 
the  usefulness  of  IOT&E  test  results  to  the  acquisition 
decision-making  process? 

These  are  the  questions  that  this  research  effort 
will  attempt  to  answer.  The  next  chapter  will  discuss  the 
evolution  of  OT&E  and  describe  how  the  test  and  evaluation 
process  enters  into  the  system  acquisition  process  in  the 
Air  Force  and  is  an  integral  part  of  it.  This  discussion 
will  provide  the  reader  with  a  better  understanding  of  the 
focus  of  this  paper. 


II. 


Literature  Review 


The  previous  chapter  introduced  the  primary  focus 
of  this  research,  namely  the  determination  of  the  useful¬ 
ness  of  AFCC  initial  operational  test  and  evaluation 
( IOT&E)  results  in  making  production  decisions.  As  men¬ 
tioned  earlier,  IOT&E  is  an  important  phase  of  test  and 
evaluation  performed  during  the  Air  Force  system  acquisi¬ 
tion  process.  To  understand  this  concept  more  fully  it  is 
first  necessary  to  have  a  feel  for  how  the  overall  Air 
Force  system  acquisition  process  functions.  Before  dis¬ 
cussing  the  system  acquisition  process  and  how  the  differ¬ 
ent  types  of  test  and  evaluation  fit  into  it,  this  chapter 
will  review  the  evolution  of  operational  test  and  evalua¬ 
tion  in  order  to  show  how  it  has  grown  in  importance  in 
recent  years.  This  discussion  will  then  lead  into  the 
current  system  acquisition  process,  the  regulatory  guidance 
for  test  and  evaluation  and,  finally,  a  brief  description 
of  the  history  and  mission  of  the  1815th  Test  and  Evalua¬ 
tion  Squadron. 

The  Evolution  of  Operational 
Test  and  Evaluation 

The  beginning  of  operational  test  and  evaluation 
of  aircraft  and  air  weapon  systems  associated  with  them 
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can  be  traced  to  the  purchase  of  the  first  airplane  from 
the  Wright  brothers  in  1909.  After  the  Wright  brothers 
had  successfully  demonstrated  the  ability  of  their  air¬ 
plane  to  satisfy  the  contract  specifications,  the  airplane 
was  shipped  to  Fort  Sam  Houston,  Texas  where  Captain 
Benjamin  Foulois  was  tasked  to  evaluate  it  and  "teach 
himself  how  to  fly."  It  was  not  long  before  he  discovered 
operational  deficiencies.  In  the  course  of  his  evaluation 
Captain  Foulois  found  that  having  no  seat  on  the  airplane 
was  a  distinct  disadvantage  during  hard  landings  as  he  was 
frequently  being  thrown  to  the  ground.  To  correct  this 
problem  he  bolted  a  metal  tractor  seat  to  the  frame  and 
while  this  helped  some,  he  still  fell  off  on  occasion. 

He  eventually  found  that  by  looping  his  Sam  Browne  belt 
through  the  seat  he  could  secure  himself  to  the  airplane 
and  solve  the  problem  (8:2-1).  This  process  of  assessing 
the  capabilities  of  a  system,  uncovering  its  deficiencies 
and  correcting  them  is  still  a  part  of  OT&E  today. 

Although  the  need  for  operational  testing  was 
recognized  early  on,  it  was  not  until  1948  that  a  system¬ 
atic  approach  to  operational  testing  of  Air  Force  weapon 
systems  was  instituted.  During  this  time  frame  the  Air 
Proving  Ground  Command  (APGC)  was  established  to  conduct 
operational  test  and  evaluation  on  Air  Force  systems. 

From  1948  until  1957  the  APGC  served  as  the  Air  Force's 


independent  testing  agency,  reporting  test  results  directly 


to  the  Chief  of  Staff  of  the  Air  Force  (8:2-1) . 

With  growth,  and  because  of  its  high  visibility,  the 
Air  Proving  Ground  Command  became  vulnerable.  Its  size 
made  it  cumbersome  and  slow  to  respond  to  developers, 
to  the  operating  commands,  and  to  the  Chief  of  Staff 
in  his  decision  making  capacity.  New  weapon  systems 
were  often  developed,  produced,  and  operationally 
deployed  before  the  Air  Proving  Ground  Command  com¬ 
pleted  its  suitability  testing.  In  fact,  it  was  never 
intended  that  the  work  of  the  APGC  be  used  to  support 
the  production  decision  making  process.  Rather,  its 
mission  was  to  test  new  production  weapon  systems,  to 
accurately  chart  their  capabilities,  and  prescribe  the 
procedures  for  their  most  effective  use  and  support. 
(25:5) 

As  a  result  of  criticisms  leveled  at  it,  the  APGC 
was  reorganized  and  combined  in  1957  with  the  Air  Force 
Armament  Center  and  made  subordinate  to  the  Air  Force 
Systems  Command  (then  called  the  Air  Research  and  Develop¬ 
ment  Command) .  The  resulting  Air  Force  Armament  Develop¬ 
ment  and  Test  Center  was  not  capable  of  performing  effec¬ 
tive,  independent  operational  testing.  The  primary 
emphasis  shifted  to  developmental  test  and  evaluation 
(8:2-1) . 


Operational  test  and  evaluation  then  became  the 
responsibility  of  the  operating  commands  who  were  left  to 
develop  their  own  policy  and  guidance  on  OT&E  since  the 
central  office  within  Air  Staff  established  in  1948  for 
this  purpose  was  abolished  when  APGC  was  discontinued. 
This  situation  caused  a  funding  problem  since  there  was 
no  provision  for  separate  OT&E  funds  within  the  command. 


10 


The  funds  for  OT&E  were  considered  part  of  operations  and 
maintenance  funding  and  quickly  became  the  area  most 
vulnerable  to  cuts  when  funding  was  limited.  OT&E  efforts 
were  reduced  significantly  because  of  this  funding  problem 
(25:5-6)  . 


It  soon  became  apparent  that  this  arrangement  was 
not  satisfactory  either.  Several  studies  advocating  the 
restoration  of  a  central  focal  point  at  Air  Staff  finally 
prompted  the  Air  Force  Chief  of  Staff  in  1964  to  direct 
the  establishment  of  an  office  within  the  Air  Staff  to 
provide  centralized  guidance  and  direction  to  the  frag¬ 
mented  OT&E  effort.  This  office  was  eventually  titled  the 
Deputy  Director  of  Operations  for  OT&E  and  was  responsible 
for  the  Air  Force  OT&E  Program  until  1974  (8:2-2). 

Although  a  focal  point  for  OT&E  was  once  again 
established  at  the  Air  Staff  level,  there  was  no  signifi¬ 
cant  change  in  the  way  OT&E  was  conducted.  While  intended 
to  support  the  acquisition  process,  operational  test 
results  frequently  came  too  late  to  affect  production  deci¬ 
sions.  OT&E  was  routinely  being  conducted  on  new  weapon 
systems  after  production  had  already  begun  (8:202). 

By  1970  it  seemed  clear  that  system  failures,  the  high 
cost  of  procurement,  and  the  extensive  post-production 
modification  of  systems  procured  in  the  1960s  could  be 
particularly  attributed  to  inadequate  and,  in  some 
cases,  the  complete  lack  of  operational  test  and 
evaluation  prior  to  production.  The  necessity  for 
operational  test  and  evaluation  as  early  as  possible 
in  the  acquisition  cycle  became  obvious.  (25:9) 
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The  Blue  Ribbon  Defense  Panel,  convened  by  Presi¬ 
dent  Nixon  to  study  the  systems  acquisition  process  within 
the  Department  of  Defense,  found  that  the  conduct  of  OT&E 
by  the  services  was  generally  inconsistent  and  ineffective. 
Some  of  the  major  findings  noted  in  Appendix  F  to  the 
Panel's  1  July  1970  report  include  the  following: 

OT&E  can  and  should  contribute  significantly  to  deci¬ 
sion  making  at  all  levels  of  DOD.  However,  unless  the 
process  of  acquiring  military  materiel  is  radically 
altered,  it  is  improbable  that  OT&E  can  be  done  in  time 
to  provide  "go/no-go"  recommendations  on  whether  to 
commence  production  of  operationally  configured  major 
systems . 

The  results  of  OT&E  which  has  been  accomplished  have 
not  been  adequately  made  available  to  or  used  by  DOD 
agencies  which  need  them.  There  is  no  method  of 
evaluating  and  preserving  such  information. 

Existing  ranges  and  other  facilities  have  been  mar¬ 
ginally  adequate  to  support  the  OT&E  which  has  been 
performed.  There  is  well  founded  doubt  concerning 
their  adequacy  for  OT&E  which  should  have  been  but  was 
not  performed.  There  is  serious  concern  as  to  whether 
future  requirements  for  such  ranges  and  other  facili¬ 
ties  can  or  will  be  met. 

OT&E  within  the  Services  is  done  most  effectively  when 
OT&E  organizations  report  directly  to  the  Chief  of  the 
Service,  representing  both  the  developer  and  the  user, 
but  organizationally  independent  of  both.  There  are, 
however,  considerable  forces  within  the  services  which 
resist  the  independence  of  OT&E  organizations. 


There  is  a  shortage  of  experienced  and  capable  per¬ 
sonnel  directly  involved  with  OT&E.  There  has  been 
inadequate  use  of  civilian  scientists  and  operations 
researchers/systems  analysts  in  OT&E  at  all  levels,  but 
particularly  where  operational  testing  is  actually 
being  planned,  conducted  and  analyzed. 

Conduct  of  needed  OT&E  is  being  adversely  affected  by 
inadequate  funding  and  particularly  by  the  lack  of 
budget  identity  for  OT&E  funds.  (25:10-12) 
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An  initial  DOD  response  to  these  recommendations 
was  the  establishment  of  an  office  to  oversee  OT&E  at  the 
DOD  level.  Deputy  Secretary  of  Defense  Packard  established 
the  office  of  the  Deputy  Director  for  Test  and  Evaluation 
(DDT&E )  within  the  office  of  the  Director  of  Defense 
Research  and  Engineering  (DDR&E)  for  this  purpose  (8:2-3). 
In  addition,  the  DOD  published  DOD  Directive  5000.1,  dated 
31  July  1971,  on  the  acquisition  of  major  defense  sy.  ,ems. 
This  directive  outlined  the  major  decision  points  in  the 
DOD  acquisition  process  and  established  the  Defense  Systems 
Acquisition  Review  Council  (DSARC).  The  DSARC  had  the 
responsibility  for  reviewing  major  programs  at  each  of 
the  three  major  decision  points  in  the  acquisition  process 
and  making  recommendations  to  the  Secretary  of  Defense  on 
required  actions.  DOD  Directive  5000.1  did  not  address 
direct  user  involvement  prior  to  the  second  major  decision 
point  (Full-Scale  Development) ,  and  it  did  not  require  an 
independent  operational  test  agency.  It  did,  however, 
stress  the  importance  of  an  operational  assessment  prior  to 
production  decisions  (2:6). 

Formal  DOD  policy  on  OT&E  was  finally  provided  on 
.i  9  January  1973  through  DOD  Directive  5G00.3,  Test  and 
L'/aiuation.  In  this  directive  OT&E  was  extensively 
defined  and  much  of  the  existing  policy  on  OT&E  confirmed. 
The  most  significant  provision  in  the  directive  required 
that  "one  major  field  agency  (or  a  limited  number  of  such 


major  field  agencies)  separate  and  distinct  from  the 
developing/procuring  command"  be  established.  In  compli¬ 
ance  with  this  direction,  the  Air  Force  established  the 
Air  Force  Test  and  Evaluation  Center  (AFTEC)  as  a  separate 
operating  agency  on  1  January  1974  (2:6-7). 

The  charter  regulation  for  AFTEC,  AF  Regulation 
23-36,  particularly  addressed  many  of  the  criticisms  made 
bv  the  Blue  Ribbon  Defense  Panel.  AFTEC  was  tasked  with 
providing  the  results  of  evaluations  on  major  systems  to 
the  Secretary  of  the  Air  Force  and  the  Chief  of  Staff  for 
consideration  in  DSARC  reviews  at  key  decision  points.  It 
established  AFTEC  as  a  separate  operating  agency  indepen¬ 
dent  from  the  developer  and  the  user.  Test  teams  were  to 
be  comprised  of  specialists  from  the  using  and  supporting 
commands  with  skills  similar  to  those  expected  by  operators 
and  maintainers  of  the  employed  system.  In  addition,  it 
provided  for  the  establishment  and  maintenance  of  a  data 
bank  to  preserve  relevant  information  on  test  and  evalua¬ 
tion  results,  to  project  testing  requirements  and  resources 
and  to  track  testing  milestones  (7;  25:26-27). 

Test  and  Evaluation  policy  did  not  significantly 
change  for  several  years. 

DODD  5000.1  went  unchanged  for  nearly  4^  years,  while 
5000.3  experienced  only  minor  changes.  The  latter 
deleted  the  option  of  having  more  than  one  independent 
agency  and  further  stipulated  that  the  one  remaining 
field  agency  would  be  independent  of  the  using  command 
as  well  as  the  developing  command.  (2:8) 


Major  changes  in  these  documents  took  place  after  the 
Office  of  Management  and  Budget  (OMB)  Circular  A-109, 
titled  Major  System  Acquisitions,  was  published  on  5  April 
1976.  The  purpose  of  the  circular  was  to  update  DOD  policy 
on  acquisition  management.  Some  of  the  significant  changes 
included  a  greater  emphasis  on  initial,  critical  phases  of 
acquisition;  the  focusing  of  acquisition  requirements  on 
user  established  mission  needs;  insistence  that  service 
requirements  be  expressed  in  mission  terms  rather  than 
known  solutions;  and  the  introduction  of  a  new  key  decision 
point.  Milestone  0,  Program  Initiation  (8:6-1).  The  Mile¬ 
stone  0  decision  point  required  the  review  and  validation 
by  DOD  decision  makers  of  the  operational  need  prior  to  the 
start  of  a  program.  The  updated  version  of  DOD  Directive 
5000.1,  published  in  1977,  incorporated  this  new  policy 
guidance.  The  directive  described  the  role  of  the  program 
manager  in  great  detail  and  deleted  the  requirement  for 
presenting  operational  test  results  to  the  DSARC  in  support 
of  production  decisions.  The  review  level  for  major  pro¬ 
grams  reverted  to  the  Service  SARC  (AFSARC)  chaired  by 
the  Service  Secretary  or  Assistant  Secretary.  Test  results 
were  only  reported  to  the  DSARC  if  deemed  necessary  by  the 
Service  Secretary  (2:9). 

The  revision  of  DODD  5000.3  contained  major  policy 
changes  with  regard  to  test  and  evaluation.  Test  objec¬ 
tives  and  evaluation  criteria  had  to  be  established  prior 


to  the  start  of  testing.  IOT&E  was  to  begin  as  early  as 
the  Conceptual  Exploration  phase  following  Milestone  0  in 
order  to  assess  the  operational  impact  of  the  alternative 
technical  approaches.  In  addition,  the  directive  required 
test  agencies  such  as  AFTEC  to  monitor  DT&E  to  insure  that 
the  system  was  ready  for  operational  testing.  The  timely 
preparation  of  the  Test  and  Evaluation  Master  Plan  (TEMP) 
was  emphasized.  It  had  to  be  completed  shortly  after 
Milestone  I  during  the  Demonstration  and  Validation  phase 
rather  than  during  the  Full-Scale  Development  phase  prior 
to  Milestone  III  (2:13).  All  of  these  changes  served  to 
bring  OT&E  into  the  acquisition  process  earlier  and  linked 
it  closely  to  the  decision-making  process. 

The  sentiments  expressed  in  the  following  quote  are 
just  as  true  today  as  they  were  over  ten  years  ago  when 
this  was  written. 

The  test  and  evaluation  evolution  we  are  witnessing 
today  is  not  unlike  the  wave  of  consumerism  which  has 
been  sweeiping  our  country  for  the  past  several  years. 
The  motivations  are  the  same--Congress,  representing 
the  people  of  our  country  is  tired  of  buying  systems 
which  cost  too  much  and  won't  perform  as  advertised. 
Therefore,  in  addition  to  continuing  emphasis  on  pro¬ 
gram  cost  and  schedule  considerations,  test  and  evalua¬ 
tion  must  take  on  new  importance.  (1:30) 

Test  and  evaluation  is  taking  on  new  importance  even  today. 

In  fact,  just  this  April  Mr.  John  E.  Krings  was  appointed 

as  the  first  director  of  the  Defense  Department's  new 

Operational  Test  and  Evaluation  Office  (19:18).  This 
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office  was  established  by  congressional  direction  for  the 
purpose  of  overseeing  OT&E  conducted  by  the  Services  on 
major  weapon  systems.  It  is  significant  to  note  that  this 
appointment  generally  lags  the  initial  ratification  of  the 
position . 

The  position  was  established  by  amendment  to  Title  10 
of  the  United  States  Code  in  the  Department  of  Defense 
Authorization  Act  of  Fiscal  Year  1984  (Public  Law  98- 
94),  dated  September  24,  1983.  The  legislative  amend¬ 
ment  to  establish  the  position  was  passed  91  to  five 
in  the  Senate  and  by  voice  vote  in  the  House  of  Repre¬ 
sentatives  implying  broad  consensus  and  no  significant 
opposition.  The  legislation  was  passed  in  spite  of 
opposition  from  the  Department  of  Defense.  (14:7) 

The  motivation  for  creating  this  new  office  can  be 
traced  back  to  the  findings  of  the  Blue  Ribbon  Defense 
Panel;  the  central  issue  being  the  independence  of  the 
OT&E  agencies  from  the  developer  and  the  user.  The  pre¬ 
vious  office  responsible  for  OT&E  at  the  DOD  level  was 
locaced  within  the  office  of  the  Under  Secretary  of  Defense 
for  Research  and  Engineering.  Having  the  OT&E  office 
within  the  office  responsible  for  development  and  procure¬ 
ment  of  weapon  systems  unavoidably  created  conflict  of 
interest.  An  explanation  of  the  reluctance  to  establish 
an  independent  OT&E  office  at  DOD  can  be  described  in 
terms  of  this  conflict. 

Operational  testing,  especially  "independent"  opera¬ 
tional  test  and  evaluation  lacks  a  natural  constitu¬ 
ency  in  the  Department  of  Defense  and  military  services 
as  development  agencies  and  using  commands  are  inter¬ 
ested  in  fielding  weapon  systems  as  soon  as  possible, 
while  operational  test  and  evaluation  agencies  identify 
deficiencies  that  impede  the  procurement  schedule. 
(14:15) 
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The  charter  for  the  Director  Operational  Test  and 
Evaluation,  Department  of  Defense  Directive  5141.2,  was 
signed  by  Deputy  Secretary  of  Defense  Taft  on  2  April  1984 
(14:32).  It  requires  the  Director  OT&E  to  be  a  permanent 
member  of  the  Defense  Review  Board  and  the  Defense  Systems 
Acquisition  Review  Council.  He  is  also  required  to  submit 
reports  of  operational  tests  of  major  weapon  system  pro¬ 
grams  to  the  Armed  Services  and  Appropriations  Committees 
on  the  Senate  and  House  of  Representatives  (14:33).  These 
provisions  will  ensure  that  operational  testing  receives 
greater  consideration  in  making  acquisition  decisions 
within  the  DOD.  In  fact,  the  Director  of  OT&E  must  approve 
the  adequacy  of  test  plans  for  major  acquisition  programs 
prior  to  implementing  them  and  submit  a  report  to  the  Secre¬ 
tary  of  Defense  and  Congress  prior  to  a  final  decision 
within  DOD  to  proceed  with  a  major  weapon  system  beyond 
low-rate  initial  production  (14:36-37). 

The  Director  and  his  staff  are  independent  of  the 
office  of  the  Under  Secretary  of  Defense  for  Research  and 
Engineering  and  report  directly  to  the  Secretary  of  Defense. 
Their  independence  is  further  assured  by  the  provision  for 
a  separate  funding  statement  in  the  Presidential  budget 
(14:36).  This  independence  is  in  consonance  with  the 
recommendation  originally  made  by  the  Blue  Ribbon  Defense 
Panel  to  have  OT&E  agencies  independent  from  the  developer 
and  user.  The  impact  this  new  office  will  have  on  the 


acquisition  decision-making  process  is  still  a  matter  of 
speculation.  The  Director  of  Operational  Test  and  Evalua¬ 
tion  has  the  potential  for  being  very  influential  in  the 
acquisition  process.  How  influential  will  depend  on  the 
Secretary  of  Defense. 

Test  and  Evaluation  in  the 
System  Acquisition  Process 

The  Air  Force  system  acquisition  process  is  the 
sequence  of  activities  that  are  followed  by  the  Air  Force 
in  order  to  acquire  a  new  system  designed  to  satisfy  valid 
mission  needs.  The  structure  of  the  current  system  acquisi¬ 
tion  process  is  based  on  the  Office  of  Management  and 
Budget  Circular  No.  A-109  that  established  the  policies  to 
be  followed  by  executive  branch  agencies  in  the  acquisition 
of  major  systems.  Department  of  Defense  Directive  (DODD) 
5000.1,  Major  System  Acquisitions ,  and  Department  of 
Defense  Instruction  (DODI)  5000.2,  Major  System  Acquisition 
Procedures ,  were  updated  to  incorporate  the  policy  guidance 
of  0M3  Circular  No.  A-109.  Together  these  three  documents 
yuide  the  system  acquisition  process  for  the  Air  Force  as 
well  as  other  military  services  and  DOD  agencies. 

The  current  system  acquisition  process  consists  of 
four  major  phases  and  four  associated  key  decision  points 
or  milestones.  While  most  major  programs  progress  through 
these  four  phases,  the  intent  of  Circular  No.  A-109  was  not 
to  establish  an  inflexible  set  of  sequenced  activities,  but 


19 


rather  to  provide  sufficient  guidance  to  allow  the  tailor¬ 
ing  of  an  acquisition  strategy  to  each  individual  program 
(26) .  With  this  in  mind,  the  following  discussion  will 
describe  the  four  phases  and  four  milestone  points  that 
are  typically  encountered  in  the  major  system  acquisition 
process.  Since  this  paper  is  primarily  concerned  with  the 
Air  Force  acquisition  process,  the  discussion  will  focus 
on  the  Air  Force  implementation  of  the  DOD  guidance  pro¬ 
vided  for  acquiring  major  systems. 

Milestone  0/Concept  Exploration .  The  first  mile¬ 
stone  point  is  sometimes  referred  to  as  Milestone  0  or  the 
Mission  Need  Determination/Program  Initiation  decision 
point.  It  is  at  this  point  that  an  operational  need  is 
identified  and  validated  based  on  an  assessment  of  the 
throat.  Operational  needs  are  initially  identified  at  the 
major  command  level  and  submitted  to  Headquarters  Air  Force 
(HQ  I'SAF)  for  validation  after  they  have  been  reviewed  by 
Air  Force  Systems  Command  (AFSC) ,  Air  Force  Logistics  Com¬ 
mand  (AFLC ) ,  and  Air  Training  Command  (ATC) ,  and  other  com¬ 
mands  as  necessary.  These  reviewing  commands  "assess  the 
technology  and  constraints  to  satisfy  the  need,  identify 
known  solution  candidates,  and  estimate  necessary  resources 
lor  need  satisfaction"  (18:5).  Validation  of  the  stated 
need  initiates  the  acquisition  process  and  begins  the 
Concept  Exploration  phase.  HQ  USAF  provides  formal 
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guidance  to  the  implementing  and  participating  commands 
by  means  of  a  document  called  the  Program  Management  Direc¬ 
tive  (PMD)  which  will  be  continually  updated  throughout 
the  system  acquisition  life  cycle  to  reflect  changes  in 
the  program. 

Before  proceeding  to  the  Concept  Exploration  phase, 
the  operational  need  is  categorized  based  on  the  type  of 
system  required  to  satisfy  it.  There  are  essentially  three 
types  of  program  categories:  DOD  Major  Programs,  Air  Force 
Designated  Acquisition  Programs,  and  Air  Force  Non-Major 
Programs.  The  program  designation  determines  the  appropri¬ 
ate  review  authority  and  the  nature  of  the  documentation 
that  is  required.  The  first  of  these,  DOD  Major  Programs, 
are  so  designated  when  the  program  has  Secretary  of  Defense 
incorest ;  involves  a  joint  acquisition  of  a  system  by  the 
DOD  and  representatives  of  another  nation,  or  by  two  or 
more  DOD  components;  the  estimated  costs  exceed  $200  mil¬ 
lion  (FY80  dollars)  in  Research  Development  Test  and  Evalu¬ 
ation  (RDT&E)  funds  or  $1  billion  (FY80  dollars)  in  pro- 
curemont/pi ou action  funds,  or  both;  and/or  the  program  has 
significant  congressional  interest  (3:6).  Currently  major 
programs  require  Secretary  of  Defense  (SECDEF )  review  at 
Milestones  I  and  II  and  will  not  enter  the  Concept  Explora¬ 
tion  phase  until  the  major  system  has  been  included  in  the 
DOD  budget  (21:11).  With  the  establishment  of  the  Defense 
Department's  new  Operational  Test  and  Evaluation  Office 
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earlier  this  year  there  is  some  indication  that  Milestone 
III  decisions  will  also  require  SECDEF  approval  in  the 
future.  The  Defense  Systems  Acquisition  Review  Council 
( DSARC )  is  the  advisory  body  that  makes  recommen  lations  to 
the  Secretary  of  Defense  on  all  such  decisions. 

The  Air  Force  Designated  Acquisition  Program 
( AFDAP )  is  the  second  program  category.  This  type  of  pro¬ 
gram  is  less  than  a  major  program  and  requires  Secretary  of 
the  Air  Force  (SAF)  approval  at  Milestones  I,  II,  and  III, 
with  the  advice  of  the  Air  Force  Systems  Acquisition  Review 
Council  (AFSARC) .  This  category  of  programs  will  usually 
have  estimated  costs  (FY80  dollars)  for  research,  develop¬ 
ment,  test  and  evaluation  between  $100  and  $200  million  or 
$500  million  and  $1  billion  for  procurement/production 
(5:96). 

The  final  program  category  is  that  of  Air  Force 
Non-Major  Programs.  Non-major  programs  are  those  which  do 
not  fall  into  one  of  the  first  two  categories.  The 
approval  level  required  is  as  directed  by  HQ  USAF  and  may 
vary  depending  on  the  program.  The  guidance  provided  by 
Air  Force  Regulation  (AFR)  800-2,  Acquisition  Program 
Management ,  states  that  programs  other  than  DOD  Major  and 
AFDAP s  are  to  be  managed  according  to  the  guidance  in 
DODD  5000.1  and  DODI  5000.2  to  the  maximum  possible  extent. 
For  the  most  part  the  management  of  these  programs  is  left 
to  the  implementing  commands  which  are  required  to  document 
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milestone  decisions  and  submit  an  information  copy  of  such 
documentation  to  HQ  USAF.  All  of  the  programs  studied  in 
this  paper  will  fall  into  this  final  category.  Non-major 
programs  have  not  received  the  close  scrutiny  that  some 
of  the  major  programs  have  because  of  the  dollar  amounts. 

However,  the  efficient  management  of  these  non-major  pro¬ 
grams  is  of  concern  also  since  there  are  potential  areas 
to  save  money. 

After  the  program  category  has  been  identified 
and  the  program  has  been  approved  at  HQ  USAF,  it  proceeds 
into  the  Concept  Exploration  phase.  During  this  phase 
there  is  only  a  commitment  to  identify  and  explore  alterna¬ 
tive  solutions  that  could  possibly  satisfy  the  operational 
need.  These  alternatives  are  usually  in  the  form  of  paper 
studies;  however,  limited  experiments  and  tests  may  be  con¬ 
ducted  "to  determine  the  technical  feasibility  of  concepts, 
defined  subsystems,  and  key  components"  (21:16).  The  test¬ 
ing  performed  during  this  phase  is  primarily  Development 
Test  and  Evaluation  and  is  normally  performed  by  the  con¬ 
tractor  submitting  a  proposal  for  consideration  in  meeting 
the  operational  need.  Even  so,  DODD  5000.3  states  that 
Operational  Test  and  Evaluation  "will  be  accomplished  as 
appropriate,  to  assess  the  operational  impact  of  candidate 
technical  approaches  and  to  assist  in  selecting  preferred 
alternative  system  concepts"  prior  to  the  Milestone  I 
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decision  (4:3).  This  implies  that  OT&E  could  be  intro¬ 
duced  as  early  as  the  Concept  Exploration  phase. 

The  program  manager  (PM)  is  assigned  during  the 
Concept  Exploration  phase.  This  individual  is  the  single 
manager  for  the  program  and  is  responsible  for  assembling 
a  team  to  help  in  accomplishing  the  program  objectives. 

At  the  end  of  the  Concept  Exploration  phase  the  PM  and 
his  management  team  will  have  selected  the  concepts  that 
were  evaluated  as  most  promising  and  which  are  felt  to 
warrant  system  demonstration.  For  major  programs  the 
Secretary  of  the  Air  Force  will  request  SECDEF  approval 
at  Milestone  I,  also  known  as  the  Requirement  Validation 
decision  point,  for  entry  of  the  program  into  the  Demon¬ 
stration  and  Validation  phase  (21:17).  Non-major  programs 
require  HQ  USAF  approval  or  approval  as  directed  by  HQ 
USAF ,  which  may  be  at  the  MAJCOM  level. 

Milestone  I /Demonstration  and  Val idation.  At  the 
Requirement  Validation  decision  point  the  documentation 
submitted  for  SECDEF  review  must  contain  the  Test  and 
Evaluation  Master  Plan  (TEMP)  as  described  in  DODD  5000.3 
(3:7).  The  TEMP  is  required  to  relate  test  objectives  to 
required  system  characteristics  and  critical  issues,  and  to 
integrate  objectives,  responsibilities,  resources  and 
schedules  for  all  test  and  evaluation  to  be  accomplished 
during  the  acquisition  process  (4:7).  The  intent  of  the 


guiding  regulations  is  clearly  to  introduce  test  and  evalu¬ 
ation  as  early  as  possible  into  the  system  acquisition 
process.  Insight  into  the  reasoning  behind  this  is  pro¬ 
vided  by  McCarty  in  the  following  narrative  on  the  purpose 
for  test  and  evaluation  in  the  acquisition  process. 

The  primary  purpose  of  test  and  evaluation  (T&E)  during 
the  acquisition  process  is  the  reduction  of  risk, 
either  the  risk  that  the  system  or  equipment  will  not 
meet  performance  specifications  or  the  risk  that  the 
system  or  equipment  cannot  be  effectively  employed  in 
its  intended  operational  environment.  Furth  rmore, 

Ti. E  is  the  primary  means  by  which  achievement  of  pro¬ 
gram  objectives  is  demonstrated  to  continuing  or 
increasing  the  commitment  of  resources  to  acquisition 
programs.  (21:27) 

Introducing  test,  and  evaluation  as  early  as  possible  into 
the  acquisition  process  allows  key  decision  makers  to 
have  a  means  by  which  they  can  judge  the  risks  inherent  in 
the  program  and  a  basis  upon  which  to  base  their  decisions. 

For  programs  that  have  been  approved  at  Milestone  I, 
the  next  phase  is  the  Demonstration  and  Validation  phase. 

The  objective  during  this  phase  if  to  validate  the  alterna¬ 
tive  solutions  and  determine  the  technical,  cost,  support- 
ability,  and  schedule  risks  involved  with  each  resulting  in 
a  selection  of  one  of  the  competing  systems  for  further 
refinement  in  the  Full-Scale  Development  phase.  The 
alternative  solutions  selected  during  concept  exploration 
are  more  clearly  defined  and  expanded  in  one  of  three  ways: 
(1)  primary  system  hardware  prototyping,  (2)  "paper” 
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studies,  or  (3)  paper  definition  plus  subsystem  proto¬ 
typing  (21:22) . 

The  prototype  systems  designed  during  this  phase 
are  primarily  concerned  with  validating  the  performance 
objectives  rather  than  the  operational  objectives  (23) . 
Validating  performance  objectives  is  a  major  concern  of 
DT& E  and  is  normally  performed  by  the  contractor  and 
selected  personnel  from  the  implementing  command  (usually 
AFSC ) .  DT&E  conducted  at  this  juncture  is  designed  to 
identify  the  preferred  technical  approach,  including  the 
identification  of  technical  risks  and  feasible  solutions 
(4:2).  Although  performance  objectives  are  the  primary 
concern  during  this  phase,  operational  objectives  need  to 
be  considered  as  well.  According  to  DODD  5000.3  OT&E 
should  be  accomplished,  as  necessary,  to  examine  the  opera¬ 
tional  aspects  of  the  selected  alternative  technical 
approaches  and  to  estimate  the  potential  operational  effec¬ 
tiveness  and  suitability  of  the  candidate  systems.  In 
addition,  decisions  made  at  this  point  "to  commit  funds 
for  production  long  lead  time  items  or  limited  production 
must  be  supported  by  OT&E  results"  (4:3-4).  The  intent 
io  clearly  to  link  the  test  and  evaluation  results  to  the 
decision-making  process.  The  implication  is  that  decisions 
made  on  the  basis  of  objective  test  results  will  be  better 
decisions.  Few  would  argue  with  this  statement. 
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Test  results  provide  valuable  information  to  pro¬ 
gram  decision  makers  for  use  in  determining  whether  the 
system  is  ready  to  proceed  to  the  next  phase  of  the  acquisi¬ 
tion  process.  When  major  test  objectives  have  not  been 
met,  chances  are  that  the  system  will  require  further 
refinement  before  it  will  be  suitable.  If  test  results 
indicate  that  the  system  is  not  able  to  meet  the  opera¬ 
tional  need  for  which  it  was  designed,  the  best  course  of 
action  may  well  be  to  reject  that  concept  and  choose 
another  that  is  more  capable  of  satisfying  the  stated 
requirement.  Testing  results  can  therefore  be  essential 
for  making  appropriate  and  informed  decisions  on  the  pro¬ 
gram. 

Milestone  II /Full-Scale  Development .  At  the  end 
of  the  Demonstration  and  Validation  phase  for  major  pro¬ 
grams  the  Secretary  of  Defense  will  decide  which  of  the 
possible  alternatives  will  proceed  into  the  Full-Scale 
Development  phase.  This  decision  point  is  known  as  Mile¬ 
stone  II  or  the  Program  Go-Ahead  decision  point.  Once  an 
alternative  has  been  chosen,  it  will  be  more  thoroughly 
developed  and  tested  during  the  Full-Scale  Development 
phase.  The  end  result  of  this  phase  will  generally  be  a 
pre-production  prototype  and  the  technical  documentation 
necessary  to  produce  the  system.  In  fact,  "the  items 
costed  must  be  sufficiently  representative  of  the  expected 


production  items  to  ensure  that  a  valid  assessment  can  be 
made  of  the  system  to  be  produced"  (4:4) . 

It  is  during  the  Full-Scale  Development  phase  that 
OT&E  is  emphasized.  At  this  point  there  is  actually  a 
functioning  system  that  the  operators  can  put  their  hands 
on.  Operational  testing  performed  on  actual  equipment 
yields  much  more  useful  and  valid  results  than  would  be 
possible  on  purely  conceptual  models  or  through  strictly 
DT&E  testing  alone.  This  fact  is  recognized  by  DODD  5000.3 
which  provides  that  the  Initial  Operational  Test  and 
Evaluation  (IOT&E)  of  the  system  must  be  accomplished 
prior  to  the  Milestone  III  decision  on  items  sufficiently 
representative  of  the  expected  production  items  to  provide 
a  valid  estimate  of  the  system's  operational  effectiveness 
and  suitability. 

It  is  important  to  realize  that  both  DT&E  and 

OT&E  testing  are  necessary  to  fully  evaluate  a  new  system. 

DT&E  alone  is  not  a  suitable  substitute  for  OT&E. 

DT&E  can  supply  useful  information  to  OT&E,  but  it 
cannot  replace  OT&E  at  any  point  in  the  life  cycle  of 
the  system.  If  an  attempt  is  made  to  dispense  with 
OT&E,  as  often  occurs  when  budget  constraints  must  be 
contended  with,  unpleasant  surprises  are  in  store  for 
the  system  operators  when  the  system  is  deployed  and 
begins  actual  operation.  (24:6-7) 

The  objective  of  OT&E  is  to  ensure  that  the  system 
to  be  produced  is  actually  the  system  that  the  user  needs. 
To  do  this  effectively  OT&E  must  be  accomplished  objec¬ 
tively  and  impartially.  The  regulatory  guidance  fosters 


impartiality  by  specifying  that  operational  testing  for 
each  DOD  component  will  be  performed  by  one  major  field 
agency,  separate  and  distinct  from  the  materiel  developing/ 
procuring  agency  and  from  the  using  agency,  and  that  the 
test  results  and  independent  evaluation  of  the  system  under 
test  will  be  directly  submitted  to  the  Military  Service 
Chief  or  Defense  Agency  Director  (4:3).  The  separate 
agency  that  is  responsible  for  operational  testing  in  the 
Air  Force  is  the  Air  Force  Operational  Test  and  Evaluation 
Center  (AFOTEC)  located  at  Kirtland  AFB ,  New  Mexico.  For 
major  programs  AFOTEC  manages  the  test  effort  and  fur¬ 
nishes  information  to  the  Air  Force  Chief  of  Staff,  the 
Secretary  of  the  Air  Force  and,  in  turn,  the  Office  of  the 
Secretary  of  Defense  and  the  Congress  on  the  results  of 
operational  testing.  For  non-major  programs  AFOTEC  usually 
serves  in  an  advisory  role  to  the  designated  operating 
command  that  was  listed  as  the  responsible  test  organiza¬ 
tion  in  the  PMD .  AFOTEC  reviews  and  approves  the  test 
plan  and  offers  assistance  as  requested  by  the  major  com¬ 
mand  throughout  the  testing  process  (6:8-2).  Operational 
Test  and  Evaluation  decisions  are  the  only  decisions  not 
under  the  purview  of  the  program  manager  (5:3). 

Milestone  II I /Product ion  and  Deployment .  The 


Milestone  III  decision  approval  is  an  actual  commitment  to 
production.  A  contract  is  negotiated  and  the  Production 


phase  begins.  The  Milestone  III  decision  must  consider 
operational  testing  results  as  well  as  cost,  schedule,  per¬ 
formance  and  supportability  considerations.  The  results 
of  the  operational  testing  are  therefore  written  to  be 
useful  to  the  program  decision  makers.  If  no  major  prob¬ 
lems  have  been  identified  during  testing,  then  the  final 
decision  will  rely  primarily  on  other  factors.  Testing 
results  have  the  greatest  impact  when  they  report  that  the 
system  tested  is  not  operationally  effective  or  suitable 
for  meeting  the  mission  need.  In  such  cases,  the  testing 
results  often  become  the  primary  factor  in  deciding 
whether  the  program  will  proceed  into  the  Production  and 
Deployment  phase. 

The  Deployment  phase  begins  when  production  items 
have  been  provided  to  the  field  for  operational  use  and  the 
using  command  becomes  responsible  for  the  equipment.  Dur¬ 
ing  this  transition  Program  Management  Responsibility 
Transfer  (PMRT)  takes  place.  This  is  when  the  overall 
responsibility  for  managing  the  program  transfers  from  AFSC 
t o  AFLC  to  include  engineering  responsibility. 

Although  it  may  seem  that  there  is  no  additional 
need  for  testing  beyond  this  point,  this  definition  is  not 
the  case. 

OT&E  attempts  to  determine  the  performance  of  a  sys¬ 
tem  under  the  most  current  operational  conditions. 

It  is  thus  a  dynamic  process,  since  the  operational 
environment,  including  the  planned  missions  and  uses 
of  a  system,  is  continually  changing,  as  are  the 
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characteristics  of  the  system  itself,  and  new  uses 
are  continually  being  developed  for  old  systems. 

OT&E  must  therefore  extend  over  the  entire  life  cycle 
of  a  system,  from  the  initial  conceptual  stages 
through  system  design  and  deployment  to  the  extension 
of  the  life  of  old  systems  by  adapting  them  to  new 
ones.  (24:2) 


This  continual  process  of  operational  testing  is  depicted 
in  Figure  1  which  shows  the  different  types  of  test  and 
evaluation  and  how  they  relate  to  the  Air  Force  major  sys¬ 
tem  acquisition  process.  Because  non-major  programs  do 
not  always  progress  through  each  of  the  major  system 
acquisition  phases,  they  are  not  handled  in  exactly  the 
same  manner.  It  is  therefore  beneficial  to  review  the 
regulatory  guidance  and  associated  documentation  require¬ 
ments  for  non-major  programs. 


Regulatory  Guidance  and 
Non-Major  Programs 

As  was  shown  earlier,  the  trend  over  the  past 
several  years  has  been  to  make  OT&E  a  much  more  prominent 
and  integral  part  of  the  acquisition  decision-making  pro¬ 
cess.  DOD  Directive  5000.1  formalizes  this  concept. 

Throughout  the  acquisition  process,  emphasis  shall  be 
placed  upon  verifying  actual  performance  through  T&E. 
The  procedures  of  DoD  Directive  5000.3  will  be  inte¬ 
gral  to  all  systems  acquisition  planning  and  decision¬ 
making.  (3:7) 

For  major  programs  the  method  for  integrating  T&E  into  the 
decision-making  process  is  to  provide  OT&E  results  for 
consideration  in  the  decisions  at  «_..~h  of  the  major 
acquisition  milestones.  This  is  accomplished  in  accordance 
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with  DOD  Instruction  5000.2  which  describes  the  primary 
documentation  requirements  for  each  decision  point.  These 
documents  incorporate  all  the  critical  factors  in  addition 
to  the  test  results  which  may  impact  the  determination  of 
whether  or  not  a  program  is  ready  to  proceed  to  the  next 
acquisition  phase.  Non-major  programs,  however,  seldom 
progress  through  all  of  the  phases  of  the  acquisition  pro¬ 
cess  and  are  frequently  only  concerned  with  the  Full- 
Scale  Development  and  the  Production/Deployment  phases. 

Due  to  the  lower  review  levels  the  documentation  described 
in  DODI  5000.2  is  not  normally  applicable  for  non-major 
programs.  Yet,  the  only  references  to  non-major  programs 
in  DODD  5000.1  and  DODD  5000.3  state  that  the  management 
principles  and  objectives  described  in  each  directive  for 
major  programs  shall  also  apply  to  programs  not  designated 
as  major  programs.  This  guidance  is  not  very  explicit. 

AF  Regulation  80-14  offers  slightly  more  guidance, 
it  states  that  reporting  requirements  for  non-major  pro¬ 
grams  are  as  specified  in  the  Program  Management  Directive 
( PMD ) ,  in  AF  Regulation  55-43,  and  in  AFR  80-14  itself. 

For  non-major  programs  the  procedures  to  follow  depend  on 
whether  the  program  was  directly  by  HQ  USAF  or  initiated  by 
the  MAJCOM.  Testing  directed  by  HQ  USAF  is  closely  moni¬ 
tored  by  AFOTEC.  The  OT&E  report  is  sent  to  the  AFOTEC 
commander  who  evaluates  the  comments  on  the  report  before 
sending  it  to  the  Chief  of  Staff  of  the  Air  Force.  For 
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testing  initiated  and  conducted  by  the  MAJCOM,  the  MAJCOM 
sends  copies  of  the  test  plan  and  test  report  to  AFOTEC 
for  entry  into  the  Air  Force  OT&E  data  bank  (11:11). 

AFOTEC  monitors  these  programs  as  well  but  to  a  much  lesser 
degree . 

It  is  interesting  to  note  that  a  study  conducted 
in  1977  on  problems  concerning  test  and  evaluation  policy 
showed  that  approximately  82  percent  of  the  programs 
assigned  to  AFOTEC  were  non-major  programs.  A  majority 
of  these  were  HQ  USAF  directed,  MAJCOM  conducted,  OT&E 
programs  that  require  close  monitoring  by  AFOTEC  (15:7-8). 
Close  monitoring  can  include  participation  in  the  test 
planning,  approval  of  the  test  plan,  participation  in  the 
test  itself,  and  commenting  on  the  test  report.  This 
represents  quite  a  work  load  to  AFOTEC.  The  contention 
made  by  the  author  of  the  study  was  that  clear,  concise, 
and  accurate  policy  directives  and  guidance  would  allow 
for  significantly  less  involvement  by  AFOTEC  in  the  non¬ 
major  programs  and  allow  more  time  for  the  management  of 
major  programs  (15:8).  A  review  of  the  applicable  regula¬ 
tions  suggest  that  this  may  still  be  an  area  for  further 
improvement . 

AFR  80-14  and  AFR  55-43  both  describe  the  review 
levels  required  for  major  programs  and  non-major  programs. 
However,  AFR  55-43  is  a  much  more  extensive  and  helpful 
document  for  guiding  the  overall  management  of  operational 
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test  and  evaluation  conducted  by  major  commands.  It 
describes  the  documentation  requirements  for  non-major 
programs  in  more  detail  and  provides  an  example  of  the 
basic  formats  for  each. 

Documents 

The  first  major  document  that  is  relevant  to  test 
and  evaluation  is  the  Program  Management  Directive  (PMD ) . 
This  document  governs  the  actions  and  participation  of  the 
implementing,  using,  supporting,  and  other  participating 
commands  in  the  program.  Although  preparing  the  PMD  is  the 
responsibility  of  HQ  USAF,  the  implementing  command, 

AFOTEC,  and  participating  commands  are  to  aid  in  its 
preparation.  The  portion  of  the  PMD  that  addresses  test 
and  evaluation  should  provide  the  critical  questions,  the 
areas  of  risk  and  specific  test  objectives  for  the  program. 
It  is  important  that  clear  test  objectives  be  established 
from  the  beginning.  The  designated  test  organization 
should  have  an  input  into  the  establishment  of  these  objec¬ 
tives  along  with  the  implementing  command  to  insure  that 
test  objectives  are  realistic. 

Frequently,  the  document  used  to  formalize  the  test 
objectives  and  the  scope  of  the  test  program  is  the  Test 
and  Evaluation  Master  Plan  (TEMP) .  This  document  is 
required  for  non-major  programs  designated  by  AFSC.  The 
TEMP  is  an  agreement  between  the  program  manager  and  the 


test  participants  on  the  scope  and  requirements  of  the 
program  and  the  roles  of  participants  (9:6-15).  It 
describes  the  plan  for  meeting  the  test  objectives  for  all 
test  and  evaluation  to  be  conducted  throughout  the  acquisi¬ 
tion  program.  The  TEMP  includes  an  explanation  of  how  the 
test  will  be  accomplished,  what  test  resources  and  funding 
will  be  required,  and  when  each  testing  event  will  occur 
for  each  program  milestone.  The  TEMP  is  usually  a  product 
of  the  Test  Plan  Working  Group  (TPWG) .  This  group  is 
established  by  the  program  manager  and  includes  representa¬ 
tives  from  the  program  office,  the  responsible  test  organi¬ 
zation,  operating  and  supporting  commands,  and  AFOTEC  (if 
involved)  (13:5) . 

So  far  the  documents  discussed  have  been  primarily 
concerned  with  the  overall  direction  of  the  testing  effort. 
These  documents  help  to  guide  the  preparation  of  the 
specific  test  plans  for  which  the  primary  test  organization 
is  responsible  for  preparing.  The  more  complete  the 
guidance  provided  through  the  PMD  and  the  TEMP,  the  better 
the  quality  of  the  test  plans.  For  HQ  USAF  directed, 

MAJCOM  conducted,  tests  the  MAJCOM  test  organization  is 
responsible  for  preparing  the  test  plan  and  sending  it  to 
AFOTEC  for  approval. 

It  is  the  general  responsibility  of  the  major  com¬ 
mands  to  establish  specific  procedures  required  to  imple¬ 
ment  AFR  80-14,  AFR  55-43,  and  associated  regulations,  and 


to  establish  a  command  OT&E  focal  point  (8:3-4;  10:10). 
AFCC  has  complied  with  this  guidance  through  an  AFCC 
supplement  to  add  to  the  general  guidance  in  AFR  80-14, 
AFCC  has  not  chosen  to  write  its  own  regulatory  guidance 
on  test  and  evaluation.  The  supplement  designates  the 
HQ  AFCC  Directorate  of  Test  and  Evaluation  (AFCC/XOY) 
as  the  command  office  of  primary  responsibility  for  all 
AFCC  test  and  evaluation.  The  1815th  Test  and  Evaluation 
Squadron  receives  their  tasking  from  this  office.  For 
programs  initiated  by  AFCC,  the  AFCC  program  manager  in 
conjunction  with  AFCC/XOY  determine  whether  a  TEMP  is 
required  (12:1).  In  all  cases,  unless  USAF  direction 
specifically  dictates  otherwise,  AFCC  program  managers 
will  establish  a  TPWG.  As  discussed  previously,  this 
group  normally  has  the  responsibility  for  writing  the 
TEMP.  If  a  TEMP  is  not  required,  the  TPWG  will  still 
perform  basically  the  same  function  by  determining  the 
relevant  test  objectives  and  test  criteria  and  setting 
up  a  schedule  for  the  required  test  events. 

The  guidance  for  establishing  relevant  test  cri¬ 
teria,  however,  is  not  always  clear.  If  the  operational 
and  maintenance  concepts  for  a  system  are  not  well 
defined,  the  guidance  provided  in  the  PMD  and  the  TEMP 
rnay  not  be  adequate  for  establishing  realistic  and  rele¬ 
vant  test  criteria.  AFR  55-43  addresses  this  possible 
lack  of  guidance  for  non-major  programs  and  recommends 
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that  the  test  organization  request  clarification  of  the 
operational  issues,  purpose,  scope  and  test  objectives 
from  the  tasking  office  (8:6-6).  This  guidance  is  required 
in  order  to  formulate  meaningful  OT&E  objectives  and  to 
provide  useful  results  to  program  decision  makers. 

The  test  organization  formulates  the  test  criteria 
from  guidance  in  the  PM D,  TEMP,  the  operational  and  main¬ 
tenance  concepts,  and  the  technical  specifications  for  the 
system.  If  insufficient  information  is  provided  from  these 
documents,  the  test  planner  must  seek  other  sources  of 
information  such  as  previous  test  plans  and  personal 
experience.  AFR  55-43  provides  some  suggestions  or.  how  to 
establish  test  criteria,  but  the  test  planner  must  still 
expend  considerable  effort  reviewing  all  applicable  docu¬ 
mentation  and  the  stated  test  objectives  to  formulate  test 
criteria  which  is  testable  and  relevant  to  decision  making. 
The  established  test  criteria  are  incorporated  into  the 
test  plan  and  used  to  evaluate  the  system  during  the  opera¬ 
tional  testing.  The  validity  of  the  test  criteria  will 
directly  affect  the  value  of  the  test  results  written  into 
the  report. 

For  the  most  part,  AFR  55-43  does  a  thorough  job 
of  providing  guidance  for  the  conduct  of  OT&E.  This  docu¬ 
ment  has  just  recently  undergone  revision  to  incorporate 
policy  changes  and  to  clarify  certain  procedures.  The 
28  June  1985  version  supersedes  the  former  publication  and 
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changes  it  from  an  AFM  to  an  AFR.  A  significant  change  is 
the  introduction  of  the  newly  established  senior  review 
official,  the  Director  of  Operational  Test  and  Evaluation. 
Additional  changes  include  incorporation  of  the  information 
from  the  previous  two  volumes  into  one,  revision  of  the 
formats  for  OT&E  test  plans  and  test  reports,  and 
incorporation  of  the  current  philosophy  on  evaluation  cri¬ 
teria.  The  revised  publication  was  written  with  less 
experienced  OT&E  test  personnel  in  mind  and  presents  more 
detailed  procedural  guidance  and  direction  (10:61).  The 
regulatory  guidance  discussed  directs  the  actions  of  major 
command  test  organizations  such  as  the  1815th  Test  and 
Evaluation  Squadron. 


The  1815th  Test  and 
Evaluation  Squadron 

The  1815th  Test  and  Evaluation  Squadron  located  at 

Wr lght-Patterson  Air  Force  Base  (AFB) ,  Ohio  is  the  primary 

organization  within  Air  Force  Communications  Command  (AFCC) 

tasked  with  Operational  Test  and  Evaluation.  It  is  an 

independent  AFCC  unit  which  reports  directly  to  HQ  AFCC. 

Its  mission  is  to  accomplish  all  AFCC  led  Operational 
Test  and  Evaluation  in  accordance  with  Air  Force 
Regulation  (AFR)  80-14  and  Air  Force  Manual  (AFM)  55- 
43;  to  manage  the  worldwide  Air  Force  Technical  Evalua¬ 
tion  Program  (TEP)  resources;  to  operate  the  AFCC 
Systems  Evaluation  School  supporting  Air  Force,  Army, 
Navy,  and  the  Defense  Communications  Agency  (DCA)  tech¬ 
nical  aspects  of  telecommunications  Operational  Testing 
and  Evaluation  (OT&E) ;  to  maintain  and  operate  a  tech¬ 
nical  support  facility  for  use  in  developing  and  refin¬ 
ing  test  procedures;  and  checking  the  operational  con¬ 
dition  of  evaluation  test  equipment.  (22:1) 
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The  squadron  was  previously  designated  the  1815th 
Test  Squadron  and  was  collocated  at  Scott  AFB,  Illinois 
with  HQ  AFCC .  The  1815th  was  reorganized  and  relocated  to 
Wright-Patterson  AFB  on  June  1,  1981  in  order  to  provide 
for  additional  space  requirements.  Today  the  squadron 
employs  slightly  over  300  personnel.  Over  70  percent  of 
this  number  are  located  at  Wright-Patterson  while  the 
remainder  are  located  at  one  of  the  four  subordinate  units 
assigned  to  the  squadron.  The  squadron  has  submitted  a 
proposal  to  change  its  title  to  the  "1815th  Test  and 
Evaluation  Group"  to  reflect  the  primary  mission  and  to 
denote  the  size  of  the  organization.  This  change  has  not 
yet  been  approved  but  is  in  line  with  current  Air  Force 
thinking  as  demonstrated  by  the  recent  addition  of  "opera¬ 
tional"  to  the  title  of  the  Air  Force  Operational  Test  and 
Evaluation  Center  (AFOTEC) . 

The  main  unit  at  Wright-Patterson  AFB  is  organized 
into  three  branches:  the  Operational  Test  and  Evaluation 
Management  Branch,  the  Follow-on  Operational  Test  and 
Evaluation  Operations  Branch,  and  the  AFCC  Systems  Evalua¬ 
tion  School.  The  Operational  Test  and  Evaluation  Manage¬ 
ment  Branch  is  responsible  for  performing  Qualif ication 
OT&E  and  Initial  OT&E.  Due  to  the  focus  of  this  paper, 
it  was  this  branch  that  furnished  much  of  the  information 
relevant  to  this  research.  The  branch  is  organized  into 
two  sections.  The  Traffic  Control  and  Landing  Systems 
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(TRACALS)  Operational  Test  and  Evaluation  Section  performs 
AFCC  led  OT&E  on  Air  Force  TRACALS  and  Navigational  Aids 
equipment,  while  the  Communications  OT&E  Section  performs 
AFCC  led  OT&E  on  Air  Force  communications  systems  to 
include:  Satellite,  High  Frequency  Radio,  Base  Communica¬ 
tions,  AUTOSEVOCOM,  and  AUTOVON  systems  (22:1).  The  opera¬ 
tional  test  and  evaluation  reports  studied  in  this  paper 
were  prepared  by  these  two  sections. 

Of  the  slightly  over  200  personnel  assigned  to 
Wright-Patterson  AFB,  only  thirteen  are  dedicated  to  per¬ 
forming  I OT&E  and  QOT&E  taskings.  The  majority  of  the 
squadron  is  assigned  to  the  Follow-on  Operational  Test 
and  Evaluation  Operations  Branch.  This  branch  is  tasked 
with  performing  FOT&E(II).  FOT&E(II)  is  the  operational 
testing  of  equipment  (already  fielded)  throughout  the 
system's  life  to  determine 

whether  the  system  can  meet  changing  operational 
requirements,  to  refine  tactics,  techniques,  doctrine, 
and  training  programs  for  the  system,  and  to  identify 
deficiencies  and  confirm  that  they  have  been  corrected. 
(11:3) 

The  AFCC  Systems  Evaluation  School  conducts  courses  on 
evaluation  techniques  for  "selected  DOD  technicians  and 
engineers  who  operate,  maintain,  and  evaluate  Air  Force 
and  DOD  communications  systems  (22:29). 

The  four  subordinate  units  of  the  1815th  are 
detachments  that  are  specifically  assigned  to  perform  test 
ana  evaluation  for  large-scale  communications  systems  whose 
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implementation  extends  over  several  years.  Detachment  1 
is  located  at  Fort  Huachuca,  Arizona  and  is  tasked  with 
participating  in  Development  Test  and  Evaluation  (DT&E) 
and  Initial  Operational  Test  and  Evaluation  (IOT&E)  for  the 
next  generation  of  tactical  communications  equipment. 

The  unit  is  actually  under  the  operational  and  functional 
control  of  the  Air  Force  Operational  Test  and  Evaluation 
Center  (AFOTEC)  and  represents  HQ  AFCC  and  the  1815th  Test 
and  Evaluation  Squadron  in  joint  service  testing.  Detach¬ 
ment  2  is  located  at  Vogelweh,  Federal  Republic  of  Germany. 
This  detachment  supports  operational  testing  requirements 
for  the  Digital  European  Backbone  (DEB)  program  as  well 
as  other  communication  systems  in  Europe.  Detachment  3  is 
located  at  Vandenberg  AFB ,  California  and  participates  as 
an  advisor  for  IOT&E  tests  of  communications  systems  associ¬ 
ated  with  the  Peacekeeper  Missile  System  (M-X  Missile) . 

This  unit  is  functionally  and  operationally  responsible  to 
AFCTEC .  Detachment  4,  located  at  Yokota  AFB,  Japan,  is 
responsible  for  IOT&E  testing  of  communications  equipment 
associated  with  the  Japan  Reconfiguration  and  Digitization 
(JRD)  program  (22) .  IOT&E  and  QOT&E  reports  completed 
by  these  detachments  were  also  included  in  this  study  if 
they  were  published  within  the  research  time  frame. 

In  addition  to  the  1815th  TES ,  two  other  units 
within  AFCC  are  tasked  with  testing  communications  equip¬ 
ment.  The  1866th  Facility  Checking  Squadron  (FCS)  is 
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responsible  for  the  follow-on  test  and  evaluation  of 
TRACALS  equipment.  It  is  also  an  independent  unit  report¬ 
ing  directly  to  AFCC.  The  1954th  Radar  Evaluation  Squadron 
(RADES)  at  Hill  AFB ,  Utah  is  a  subordinate  unit  of  the 
Engineering  Installation  Division  of  AFCC.  It  is  respon¬ 
sible  for  all  operational  testing  of  heavy  radar  equipment. 
Although  these  two  units  are  occasionally  tasked  to  con¬ 
duct  or  participate  in  DT&E  and  OT&E  of  new  systems,  they 
actually  only  accomplish  technical  evaluations  of  perform¬ 
ance.  Evaluations  in  the  areas  of  training,  safety, 
reliability,  maintainability,  and  the  other  suitability 
factors  accomplished  in  a  full  OT&E  effort  are  not  nor¬ 
mally  performed  (16:12).  Therefore,  test  reports  prepared 
by  these  units  were  not  considered  in  this  research.  The 
next  chapter  will  discuss  the  methodology  used  to  conduct 
this  research. 
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III.  Methodolo 


The  method  employed  to  obtain  answers  to  the 
research  questions  includes  reviewing  pertinent  regula¬ 
tions  and  selected  OT&E  reports,  and  conducting  structured 
interviews  of  test  personnel  and  program  managers.  This 
method  is  deemed  appropriate  because  of  the  nature  of  the 
information  required  to  answer  the  research  questions. 

Regulations  and  documents  applicable  to  Air  Force 
test  and  evaluation  identify  the  intended  purpose  for  OT&E 
and  the  specific  procedures  to  be  followed  when  performing 
OT&E.  These  sources  provide  the  background  necessary  to 
understand  test  and  evaluation  and  provide  a  basis  upon 
which  to  formulate  relevant  interview  questions. 

Test  reports  are  reviewed  to  obtain  the  final 
results  of  the  test  and  the  recommendations  relevant  to 
the  production  decision.  Based  on  this  review  and  the 
review  of  the  applicable  regulations  and  documents  a  set 
of  questions  can  be  developed  for  conducting  interviews. 
Interviews  are  conducted  with  the  appropriate  test 
directors  and  program  managers  to  determine  their  percep¬ 
tions  of  the  test  results  as  it  is  believed  that  the 
testers  and  the  decision  makers  that  produce  and/or  receive 
copies  of  the  test  reports  are  the  ones  that  could  most 


accurately  assess  their  value.  The  testers  develop  the 
test  plan,  actually  perforin  the  testing,  and  write  up  the 
final  test  report.  The  program  managers  are  responsible 
for  the  overall  management  of  a  particular  program  and 
receive  a  copy  of  the  test  report  for  use  in  the  decision¬ 
making  process  as  intended  by  regulation.  Through  inter¬ 
views  of  test  personnel  and  program  managers  associated 
with  specific  test  reports,  a  more  complete  understanding 
of  the  usefulness  of  test  results  in  making  the  first  major 
production  decision  can  be  obtained. 

Review  of  Regulations  and 
Selected  Test  Reports 

The  regulations  and  documents  that  are  most  appli¬ 
cable  to  Air  Force  test  and  evaluation  include  DOD  Direc¬ 
tive  5000.3,  Test  and  Evaluation;  AF  Regulation  80-14, 

Test  and  Evaluation;  AF  Regulation  55-43,  Management  of 
Operational  Test  and  Evaluation;  and  AFOTEC  Regulation  55-1 
AFOTEC  Operations  Regulation .  These  documents  can  be 
reviewed  to  determine  the  intended  purpose  of  Air  Force 
test  and  evaluation,  the  procedures  and  guidelines  for  con¬ 
ducting  OT&E ,  and  the  required  contents  of  OT&E  reports. 
This  information  is  used  in  the  review  of  selected  test 
reports  and  in  the  formulation  of  interview  questions. 

A  census  of  IOT&E  and  QOT&E  test  reports  from  the 
last  four  years  are  used  for  the  purpose  of  this  research 
effort.  Over  the  past  four  years  an  average  of 


approximately  three  IOT&E/QOT&E  test  reports  have  been 
written  each  year  in  AFCC.  This  was  not  considered  a  large 
enough  population  to  facilitate  taking  a  representative 
random  sample.  Because  of  these  small  numbers  all  IOT&E 
and  QOT&E  test  reports  from  the  last  four  years  were 
reviewed  for  the  purpose  of  noting  specific  recommenda¬ 
tions  made  on  each.  The  time  frame  of  four  years  was 
chosen  because  the  1815th  TES,  which  is  the  responsible 
organization  for  performing  AFCC  operational  tests  and 
evaluations,  has  only  been  in  being  in  its  current  con¬ 
figuration  for  a  little  over  four  years.  IOT&E  and  QOT&E 
test  reports  from  previous  years  will  not  be  taken  because 
of  the  procedural  changes  associated  with  the  formation  of 
the  new  organization.  The  test  results  from  the  last  four 
years  are  believed  to  be  more  representative  of  the  cur¬ 
rent  organizational  structure  and  the  current  regulatory 
guidance.  This  approach  offers  more  validity  to  the 
research  results  so  that  they  can  be  generalized  across 
the  total  population  of  IOT&E  and  QOT&E  conducted  by  AFCC. 

Selection  of  Personnel 
to  be  Interviewed 

The  selection  of  test  personnel  and  program  mana¬ 
gers  to  be  interviewed  is  based  on  the  specific  communica¬ 
tion  system  that  was  tested.  Test  personnel  that  have  par¬ 
ticipated  in  the  testing  of  the  system  are  chosen  because 
they  are  most  able  to  recall  the  testing  procedures  and 


any  problems  encountered  during  the  testing.  It  is  likely 
that  problems  or  limitations  encountered  while  testing 
may  have  detracted  from  the  usefulness  of  the  test  results. 
Therefore,  it  is  important  to  understand  what  these  prob¬ 
lems  are  when  considering  test  results  and  the  usefulness 
of  the  test  report  in  the  decision-making  process. 

Program  managers  maintain  complete  records  of 
actions  taken  during  the  acquisition  process.  Because 
these  records  document  major  decisions,  they  can  normally 
reveal  when  the  major  production  decision  was  made  and  the 
factors  considered  in  the  decision.  This  information  is 
helpful  in  determining  the  usefulness  of  the  OT&E  test 
report  to  the  production  decision. 

Because  of  the  nature  of  military  life,  an  obvious 
limitation  to  this  selection  methodology  is  the  fact  that 
several  of  the  original  program  managers  and  test 
directors  may  no  longer  be  assigned  to  the  same  organiza¬ 
tion,  especially  for  older  programs.  Where  possible  the 
original  participants  are  interviewed.  If  the  original 
program  manager  is  no  longer  assigned  to  the  program  and 
can  not  be  reached,  then  the  current  program  manager  of 
the  program  can  be  interviewed.  Due  to  the  extensive 
records  maintained  on  each  program  it  is  believed  that  the 
current  program  manager  is  capable  of  answering  the  inter¬ 
view  questions  satisfactorily.  If  the  original  test 
director  is  no  longer  assigned  to  the  test  squadron  and 


can  not  be  reached,  then  a  test  team  participant  that  is 
still  assigned  to  the  squadron  can  be  interviewed. 


Development  of  Interview  Questions 

Interview  questions  are  structured  to  answer  the 
two  major  research  questions  presented  in  Chapter  I  and 
were  intentionally  made  open-ended  to  avoid  restriction  of 
the  answers  of  the  respondents.  Because  of  the  different 
perspectives  and  areas  of  expertise  that  the  testers  and 
program  managers  possess,  the  interview  questions  are  not 
the  same  for  both.  Testers  are  asked  questions  that  are 
more  specific  to  the  testing  process  and  which  primarily 
address  the  research  question  concerning  the  sufficiency 
of  test  and  evaluation  reports.  Program  managers  are  asked 
questions  that  are  more  specific  to  the  acquisition 
decision-making  process  and  which  primarily  address  the 
research  question  dealing  with  the  usefulness  of  test 
reports  in  making  production  decisions.  There  are  also 
interview  questions  asked  of  testers  and  program  managers 
that  seek  to  answer  both  research  questions  and  not  just 
the  one  associated  with  their  specific  areas  of  expertise. 
The  interview  questions  that  are  asked  of  testers  and 
program  managers  are  listed  in  Table  I  and  Table  II  respec- 


TABLE  I 


INTERVIEW  QUESTIONS  ASKED  OF  TESTERS 


1.  What  was  the  composition  and  qualification  of 
the  test  team? 

2.  At  what  phase  of  the  program  did  AFOTEC/1815th 
get  involved  (Concept  Exploration,  Demonstration  and  Vali¬ 
dation,  etc  - ) ? 

3.  What  kind  of  communication  did  the  test 
director  have  with  the  program  manager? 

4.  Did  the  operational  test  team  and  the  program 
manager  agree  on  operational  objectives  for  testing? 

Please  explain. 

5.  Were  the  priorities  similar  for  the  test  team 
and  the  program  manager? 

6.  Were  there  any  limitations  or  constraints 
placed  on  the  test  team?  If  so,  what  were  they? 

7.  Were  these  limitations  or  constraints  valid? 

8.  Did  these  limitations  or  constraints  adversely 
affect  the  value  of  the  test  report?  In  what  way? 

9.  How  were  the  criteria  used  for  operational 
testing  obtained/determined?  What  guidance  was  the  test 
team  provided? 

10.  Were  there  any  deficiencies  in  the  test  report 
itself?  If  so,  what  were  they? 

11.  Were  significant  results  uncovered  in  testing 
which  were  not  included  in  the  test  report?  If  so,  what 
were  they  and  why  were  they  left  out? 

12.  What  could  have  been  done  to  improve  the  test¬ 
ing  effort? 


TABLE  II 


INTERVIEW  QUESTIONS  ASKED  OF  PROGRAM  MANAGERS 


1.  What  is  the  dollar  value  of  this  program? 

2.  How  many  years  has  this  program  been  in  being 
and  how  many  more  is  it  projected  for? 

3.  Was  this  program  directed  by  HQ  USAF  or  initi¬ 
ated  by  the  major  command? 

4.  What  was  the  date  of  the  production  decision? 

5.  What  is  the  current  status  of  the  program? 

6.  Did  you  review  the  OT&E  results  such  as  the 
test  reports,  service  reports,  and  status  reports?  If  so, 
were  these  reports  complete  and  timely? 

7.  Did  the  results  of  these  reports  impact  the 
production  decision  for  the  program?  In  what  way? 

8.  Were  there  any  unusual  problems  or  limiting 
factors  during  the  operational  test  that  may  have  impacted 
the  value  of  the  tes  reports?  If  so,  what  were  they? 

9.  Did  you  feel  that  the  personnel  performing  the 
operational  testing  were  well  trained  and  qualified  to  do 
the  testing?  What  were  the  indications  that  they  were/ 
were  not? 

10.  Was  there  anything  left  out  of  the  operational 
test  report  that  you  would  have  liked  to  have  seen 
included?  Please  explain. 

11.  Do  you  have  any  recommendations  on  how  to 
improve  the  OT&E  performed  by  the  1815th? 

12.  What  were  the  factors  that  impacted  the  produc¬ 
tion  decision  for  this  program?  Which  had  the  most  impact? 

13.  Was  the  production  decision  made  prior  to  the 
completion  of  the  OT&E?  If  so,  what  was  the  reason  for 
this? 


Conducting  Interviews 

Interviews  are  conducted  in  person  whenever  pos¬ 
sible.  If  personal  interviews  are  not  feasible,  telephone 
interviews  can  be  conducted.  Personal  interviews  are  pre¬ 
ferred  because  they  allow  the  greatest  transfer  of  informa¬ 
tion,  especially  if  the  test  reports  and  program  files  are 
readily  available  for  review  and  discussion,  but  this  may 
not  be  possible  in  many  cases.  The  interview  questions 
listed  in  Tables  land  II  serve  as  the  basis  for  discus¬ 
sion.  These  questions  were  pretested  by  conducting  pre¬ 
liminary  interviews  with  the  1815th  TES  to  determine  their 
adequacy  in  obtaining  the  information  required  to  answer 
the  research  questions.  In  addition  to  the  interview  ques¬ 
tions,  a  listing  of  the  specific  recommendations  made  in 
individual  test  reports  are  used  to  guide  questions  on  how 
these  recommendations  are  used. 

Interview  results  are  reviewed  to  determine  any 
common  trends  or  specific  problem  areas  and  to  provide  the 
basis  for  conclusions.  A  complete  discussion  of  the  find¬ 
ings  is  presented  in  the  next  chapter.  Recommendations 
formulated  from  the  results  are  stated  in  the  final  chapter 
of  this  thesis. 
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IV.  Findings 


Examination  of  the  test  reports  written  by  the 
1815th  OT jE  Management  Branch  over  the  past  four  years 
revealed  that  most  of  the  tests  conducted  were  QOT&E. 
Fourteen  test  reports  were  reviewed  altogther.  Of  these, 
ten  were  classified  as  QOT&E,  three  as  IOT&E,  and  one  was 
a  special  evaluation  that  could  not  be  classified  as  either 
an  IOT&E  or  a  QOT&E.  The  special  evaluation  was  conducted 
to  assess  the  ability  of  two  existing  systems  to  interface 
with  each  other  and  did  not  incorporate  many  of  the  opera¬ 
tional  effectiveness  and  operational  suitability  considera¬ 
tions  normally  evaluated  in  an  OT&E.  Therefore,  this 
evaluation  is  not  included  as  part  of  the  results  in  this 
chapter.  Two  of  the  test  reports  designated  as  IOT&E  were 
actually  tests  conducted  on  improvements  to  existing  Air 
Force  systems.  They  were  not  classified  as  QOT&E  efforts 
because  of  the  expenditure  of  research  and  development 
f  unds . 

As  noted  earlier,  new  systems  requiring  an  exten¬ 
sive  research  and  development  effort  are  quite  often  priced 
in  the  DOD  Major  or  the  AF  Designated  Acquisition  Program 
category.  The  responsibility  for  conducting  operational 
testing  for  these  programs  is  generally  assigned  to  AFOTEC . 


Programs  normally  considered  non-major  are  those  that 
either  evaluate  off-the-shelf  equipment  for  military  use 
or  seek  to  improve  existing  equipment  in  the  Air  Force 
inventory  by  modifying  it.  These  are  the  types  of  systems 
that  the  major  commands  are  usually  directed  to  test 
because  they  seldom  cost  as  much  as  new  systems  which 
undergo  extensive  research  and  development  efforts. 

Table  III  shows  the  dollar  values  of  the  programs  that 
were  studied.  The  majority  of  the  programs  were  valued 
under  $25  million.  The  one  program  costing  $1  billion 
involves  the  procurement  of  an  off-the-shelf  system  that 
is  being  installed  Air  Force-wide  to  replace  existing 
systems  which  no  longer  provide  the  capability  required. 

All  of  the  programs  studied  were  non-major  programs 
directed  by  HQ  USAF.  A  review  of  the  regulatory  guidance 
and  associated  documentation  concerning  non-major  programs 
was  made  and  presented  in  Chapter  II.  This  information 
provided  insight  into  the  management  of  non-major  programs 
and  aided  in  the  design  of  the  interview  questions  asked  of 
test  directors  and  program  managers. 

interview  Results 

A  list  of  the  interviews  conducted  for  this 
research  is  provided  in  the  appendix.  Thirteen  program 
managers  and  eleven  test  directors  or  team  members  were 
interviewed.  One  of  the  program  managers  was  responsible 


TABLE  III 


DOLLAR  VALUES  OF  PROGRAMS  STUDIED 


Dollar  Value 

Number  of  Programs 

Less  than  $10  Million 

4 

$10  Million  to 

$25  Million 

6 

$25  Million  to 

$50  Million 

0 

$50  Million  to 

$100  Million 

2 

$1  Billion 

1 

for  managing  two  of  the  programs  studied  and  two  of  the 
test  directors  were  responsible  for  two  separate  tests 
each.  The  test  director  for  one  program  could  not  be 
reached  for  an  interview  and  no  other  team  members  with 
knowledge  of  the  OT&E  could  be  found.  Since  the  special 
evaluation  was  not  a  true  OT&E  effort,  the  responses  of 
the  test  director  and  the  program  manager  for  this  program 
have  not  been  included  in  the  following  results. 

The  results  of  the  structured  interviews  provided 
the  basis  for  determining  the  answers  to  the  two  major 
research  questions  presented  in  Chapter  I.  The  first 
research  question  was: 

Are  the  evaluation  reports  sufficient  in  terms  of 
practical  recommendations  to  provide  valid  input  into 
tne  production  decisions? 

In  terms  of  the  interview  questions,  the  ones  that 
most  apply  to  this  question  are  those  dealing  with  the 


test  report  itself,  the  qualifications  of  the  test  team, 
the  determination  of  the  test  criteria,  and  the  limitations 
of  the  te jt.  Although  the  interview  questions  that  address 
the  test  report  directly  are  most  valid,  the  other  con¬ 
siderations  are  also  valid  in  that  they  help  to  determine 
the  quality  of  the  test  report.  The  qualifications  of  the 
test  team  can  affect  the  quality  of  the  test  report  in  two 
ways,  through  the  test  plan  and  the  test  itself.  If  the 
test  team  is  well  qualified  it  will  be  able  to  construct 
an  objective  and  valid  test  plan  for  evaluation  of  the 
system.  An  important  part  of  test  planning  is  the  deter¬ 
mination  of  evaluation  criteria.  These  criteria  establish 
levels  against  which  a  system's  performance  is  compared 
in  order  to  judge  the  achievement  of  required  operational 
effectiveness  or  suitability  characteristics.  To  be 
valid  the  criteria  must  be  directly  related  to  user 
requirements.  The  value  of  the  test  report  can  be  affected 
if  the  test  team  does  not  represent  typical  operators  and 
maintainers.  The  qualifications  and  skill  levels  of  the 
test  team  must  be  representative  of  those  expected  to  main¬ 
tain  and  operate  the  system  to  ensure  that  the  test 
environment  is  as  realistic  as  possible.  Limitations  of 
the  tost  environment  determine  which  test  objectives  can 
and  can  not  be  addressed  in  the  test.  It  is  important  to 
know  what  these  are  when  evaluating  a  system. 
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The  questions  dealing  directly  with  the  program 


manager's  perception  of  the  test  reports  provided  the 
results  listed  in  Table  IV.  All  but  one  of  the  program 
managers  interviewed  felt  that  the  test  reports  were 
complete  and  provided  the  necessary  information  they 
required  for  program  decisions.  In  the  one  case  that  the 
test  report  was  not  considered  sufficient,  the  program 
manager  believed  that  the  evaluation  criteria  were  inade¬ 
quate  for  some  of  the  test  objectives.  Several  of  the 
objectives  had  incomplete  evaluation  criteria  or  no  cri¬ 
teria  at  all,  which  made  some  of  the  test  results  incon¬ 
clusive.  The  overall  test  report  was  seen  to  be  of  value, 
however,  and  was  the  primary  factor  used  in  the  decision 
to  buy  the  system. 


TABLE  IV 

PROGRAM  MANAGER'S  PERCEPTION  OF  THE  TEST  REPORT 


Response 

Questions 

Yes 

No 

Is  the  report  complete? 

12 

1 

Is  the  report  timely? 

8 

5 

The  issue  of  evaluation 

criteria,  although  it 

was 

an  area  of  concern  to  only  one  of  the  program  managers 
interviewed,  was  an  area  of  significant  concern  to  the  test 
directors.  From  the  discussion  of  evaluation  criteria 


presented  earlier  in  the  paper,  it  was  apparent  that  there 
is  no  standard  methodology  for  determining  evaluation 
criteria.  This  is  a  function  of  the  differences  in  the 
systems  tested  and  the  extent  of  the  guidance  provided  in 
the  programming  documents.  In  determining  the  relevant 
evaluation  criteria,  the  testers  usually  review  all 
applicable  program  documents,  regulatory  guidance,  and 
technical  specifications  first.  If  satisfactory  evaluation 
criteria  can  not  be  established  from  these  documents, 
further  guidance  is  requested  from  the  program  office. 

When  questioned  about  the  evaluation  criteria,  the  test 
directors  interviewed  revealed  that  the  programming  docu¬ 
mentation  did  not  provide  adequate  guidance  for  the 
establishment  of  specific  evaluation  criteria  for  six  of 
the  thirteen  programs  studied  and  that  further  guidance 
had  to  be  requested  from  the  program  office.  In  at  least 
three  of  the  cases  the  evaluation  criteria  was  the  result 
of  brainstorming  sessions  among  the  TPWG  participants. 

For  each  test  program  the  criteria  were  eventually  agreed 
upon  by  the  program  manager  and  the  test  participants 
prior  to  testing. 

As  shown  in  Table  IV,  the  primary  complaint  from 
program  managers  concerning  the  test  reports  was  the  delay 
in  receiving  them  after  completion  of  testing.  Table  V 
shows  the  time  lapses  experienced  for  the  programs 
studied.  The  turnaround  time  for  distribution  of  the  test 


TABLE  V 


TIME  BETWEEN 

TEST 

OF 

COMPLETION  AND 
TEST  REPORT 

PUBLICATION 

Length  of  Time 

Number  of  Programs 

Less 

than  1  month 

2 

1  to 

2  months 

4 

2  to 

3  months 

3 

3  to 

4  months 

1 

4  to 

5  months 

0 

5  months  or  greater 

3 

reports  for  all  the  programs  reviewed  ranged  from  two  weeks 
to  five  months.  A  majority  of  the  test  reports  were  dis¬ 
tributed  in  less  than  three  months  after  test  completion. 
Although  program  managers  expressed  dismay  at  the  delay 
in  test  report  distribution  for  five  out  of  the  thirteen 
programs,  it  is  interesting  to  note  that  for  all  but  two 
of  the  programs  the  program  manager  was  in  continual  con¬ 
tact  with  the  test  director  throughout  the  testing  period 
and  received  weekly  status  reports  of  the  progress  and  test 
results.  It  appears  as  if  the  test  reports  may  actually 
be  more  of  a  formality  for  providing  test  results  than  the 
sole  source  of  system  performance. 

When  questioned  on  whether  they  felt  the  test  team 
was  well  qualified  to  perform  the  testing,  the  program 
managers  were  unanimous  in  their  response.  They  all  felt 
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that  the  team  was  well  trained  and  qualified  to  perform 
the  testing.  In  questioning  the  test  directors  on  the 
composition  of  the  team  and  qualifications  of  the  team 
members,  it  was  evident  that  the  teams  were  carefully 
chosen.  Operations  and  maintenance  personnel  whose  train 
ing  and  skill  levels  were  representative  of  those  to  be 
assigned  to  the  system  tested  were  chosen  for  the  OT&E 
effort.  This  greatly  contributed  to  the  realism  of  the 
testing  and,  perhaps,  to  the  perception  of  the  validity 
of  the  reported  results. 

With  regard  to  the  realism  of  the  test,  the  area 
of  most  concern  was  that  of  limitations  placed  on  the 
test  environment.  Since  the  test  articles  were  normally 
production  items,  the  system  configuration  was  as  close 
to  the  actual  as  possible.  Several  of  the  tests  were  per 
formed  in  a  laboratory  environment  but  this  did  not  seem 
to  adversely  affect  the  test  results.  Although  the  test 
environment  was  not  as  realistic  as  it  would  have  been  in 
a  true  operational  setting,  extra  effort  was  expended  to 
simulate  the  operational  environments.  Laboratory  tests 
were  generally  conducted  to  test  the  interface  capability 
between  several  pieces  of  equipment  prior  to  placing  this 
equipment  in  the  field  environment.  For  those  tests  per¬ 
formed  in  an  operational  setting,  only  those  scenarios 
deemed  most  representative  of  the  operating  environment 


were  tested.  It  was  not  possible  to  exhaustively  test  all 
possible  scenarios  but  this  was  seen  as  a  valid  constraint. 

The  most  common  limitation  was  the  length  of  time 
allowed  for  testing.  Table  VI  shows  the  test  durations 
for  the  programs  studied.  Test  periods  ranged  from  two 
weeks  to  180  days.  The  tests  that  lasted  only  two  to  three 
weeks  were  normally  testing  a  relatively  simple  technology 
or  were  restricted  because  of  mission  requirements  that 
would  not  allow  further  testing  on  the  operational  system. 
The  test  duration  was  seen  as  a  limiting  factor  by  the 
test  director  primarily  because  it  often  did  not  allow 
sufficient  time  to  collect  significant  data  on  the  relia¬ 
bility  and  maintainability  of  the  system.  It  was  difficult 
to  determine  whether  the  system  failures  experienced  during 
the  test  period  would  be  representative  of  the  system  under 
continual  operation  because  of  such  a  small  sample  size. 

For  the  most  part,  however,  time  limitations  were  seen  as 
valid  by  the  testers  when  considering  the  tradeoff  between 
costs  and  the  marginal  utility  of  longer  test  periods. 

Another  factor  related  to  this  was  the  logistics 
and  supply  support  which  was  often  provided  by  the  con¬ 
tractor  of  the  system  during  the  test.  This  supply  support 
was  not  necessarily  representative  of  the  support  the  sys¬ 
tem  would  receive  under  normal  operating  conditions.  Any 
such  limitations  experienced  during  the  test  were  noted 
in  the  test  report. 
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TABLE  VI 


TEST  DURATION 


Length  of  Test 

Number  of  Programs 

Less  than  30  days 

5 

30  to  60  days 

0 

60  to  90  days 

4 

90  to  120  days 

1 

120  to  150  days 

1 

150  so  180  days 

0 

180  days  or  more 

2 

The  other  limitations  experienced  during  testing 
which  impacted  the  perception  of  the  validity  of  the 
report  were  related  to  the  readiness  of  the  system  to  be 
operationally  tested.  In  some  cases  the  technical  data 
was  incomplete  and  the  system  software  was  not  adequately 
tested  during  the  developmental  phase  prior  to  the  start  of 
operational  testing.  These  limitations  impacted  the  test 
results  but  not  the  value  of  the  test  report.  The  test 
report  served  its  purpose  by  providing  an  accurate  evalua¬ 
tion  of  the  current  system  o  nfiguration  for  use  by  the 
program  decision  makers.  The  overall  results  indicate 
that  test  reports  are  perceived  to  be  sufficient  to  provide 
valid  inputs  into  the  program  decision-making  process. 


61 


The  second  major  research  question  that  this  study 
addressed  was: 

If  the  test  results  are  deemed  sufficient,  are  they 

being  used  effectively  by  program  decision  makers? 

As  noted  above,  program  managers  view  the  test  reports 
as  being  valuable  to  the  decision-making  process.  To 
determine  how  they  are  used  in  the  decision-making  process 
the  answers  to  the  interview  questions  that  apply  to  this 
research  question  were  examined.  The  questions  that  most 
apply  are  the  ones  concerning  the  timing  of  the  production 
decision,  the  impact  of  the  test  results,  the  factors  that 
impacted  the  production  decision,  and  the  phase  the  test 
organization  became  involved  with  the  program.  The  timing 
of  the  production  decision  can  show  whether  or  not  the 
OT&E  results  were  used  primarily  because  decisions  made 
prior  to  the  start  of  testing  could  not  have  possibly 
included  the  test  results  in  the  decision  process.  The 
impact  of  test  results  deal  directly  with  how  the  test 
results  were  used.  The  factors  that  impacted  the  produc¬ 
tion  decision  other  than  testing  can  help  provide  an  under 
standing  of  the  dynamics  involved  in  the  decision-making 
process.  The  phase  the  test  organization  became  involved 
in  the  program  may  be  able  to  reveal  whether  or  not  early 
involvement  makes  a  difference  in  the  use  of  test  results 
in  the  decision-making  process. 


One  of  the  programs  studied  did  not  involve  the 
acquisition  of  equipment.  Its  purpose  was  to  assess  avail¬ 
able  technology  for  future  application.  For  this  reason 
it  will  not  be  included  in  the  results  for  the  second 
research  question.  For  six  out  of  the  twelve  programs 
involving  the  acquisition  of  equipment,  the  production 
decision  was  made  before  the  testing  began.  Because  the 
systems  tested  were  not  dealing  with  totally  new  and 
untried  technology,  the  probability  that  the  systems  would 
not  meet  the  mission  need  was  deemed  small  and  the  risk 
acceptable.  The  OT&E  test  results  had  no  impact  on  the 
production  decisions  in  these  cases.  The  OT&E  was  used 
primarily  to  identify  deficiencies  and  to  ensure  that  the 
system  would  operate  satisfactorily  under  actual  field 
conditions.  Although  the  operational  testing  had  no 
impact  on  these  production  decisions,  the  results  could 
have  possibly  led  to  the  termination  of  the  procurement  if 
in  fact  a  system  was  proven  incapable  of  meeting  the  mis¬ 
sion  need.  None  of  these  programs  was  cancelled  due  to 
operational  test  results. 

Although  the  focus  of  this  research  was  primarily 
concerned  with  the  usefulness  of  test  results  in  the  pro¬ 
duction  decision,  this  was  not  the  only  area  in  which  the 
test  reports  proved  useful  for  the  programs  studied  as 
noted  above.  In  fact,  the  operational  test  results  were 
considered  the  primary  factor  in  making  production 
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decisions  for  just  three  of  the  programs  studied  that  made 
the  production  decision  after  the  OT&E  was  completed.  Two 
of  these  programs  tested  several  competing  off-the-shelf 
systems  and  reported  the  results  of  each  system  for 
comparative  analysis.  This  analysis  aided  decision  makers 
in  determining  which  system  would  !  est  meet  the  stated 
need.  The  other  program  tested  just  one  off-the-shelf 
system  to  determine  whether  or  not  it  had  the  required 
capability  to  satisfy  the  need.  Table  VII  reflects  the 
factors,  as  reported  by  the  program  managers,  that  had  the 
most  impact  on  production  decisions. 

TABLE  VII 

FACTORS  HAVING  THE  MOST  IMPACT  ON  THE 
PRODUCTION  DECISION 

List  of  Factors  Number  of  Programs 

Mission  Need  (Downward  Directed)  6 

Performance  Specifications  (DT&E)  2 

OT&E  Results  3 

Availability  of  Funds  and  Hardware  1 

Four  out  of  the  six  programs  in  which  the  mission 
need  was  listed  as  the  factor  having  the  most  impact  on 
the  production  decision  made  the  production  decision  prior 
to  the  start  of  testing.  For  the  two  programs  listing 
performance  specifications  as  the  primary  factor  the 


results  were  split.  While  the  three  of  these  programs 
which  made  the  production  decision  after  test  completion 
did  not  list  OT&E  as  the  primary  decision  factor,  the  OT&E 
results  were  nonetheless  a  significant  factor  in  the  deci¬ 
sion. 

One  system  was  already  in  the  field  before  opera¬ 
tional  testing  was  requested.  The  primary  factors  in  the 
production  decision  were  the  availability  of  hardware  that 
appeared  to  meet  the  need  and  the  availability  of  funding 
for  immediate  use  from  another  program.  The  result  of 
this  production  decision  was  a  fielded  system  that  lacked 
adequately  trained  personnel  and  logistics  support. 

Earlier  operational  testing  is  likely  to  have  prevented 
many  of  the  problems  that  have  plagued  this  program. 

Involvement  in  the  acquisition  process  ideally 
starts  as  early  as  possible.  The  point  in  the  acquisition 
process  that  the  test  organization  became  involved  varied 
for  each  program.  For  six  of  the  programs  the  1815th  TES 
became  involved  during  the  Full-Scale  Development  phase. 
For  one  program  involvement  began  during  the  Demonstration 
and  Validation  phase.  For  the  remaining  five  programs 
involvement  began  during  the  Concept  Exploration  phase. 

It  is  interesting  to  note  that  the  programs 
involving  the  1815th  as  early  as  the  Concept  Exploration 
phase  fell  into  one  of  two  categories:  modification  and 
improvement  programs  or  high  cost  programs.  The  main 
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reason  that  modification  and  improvement  programs  involve 
the  test  organization  early  is  that  quite  often  it  is  the 
test  organization  that  initially  identifies  the  need  to 
modify  or  improve  the  system.  Three  programs  were  in  this 
category.  The  remaining  two  were  high  cost  programs. 

One  was  the  $1  billion  program  mentioned  earlier  and  the 
other  was  a  program  that  is  valued  around  $60  million. 

For  these  programs  the  test  organization  may  have  been 
involved  earlier  in  the  acquisition  process  primarily  due 
to  higher  visibility.  The  phase  of  involvement  in  the 
acquisition  process  was  not  a  good  indicator  of  whether  or 
not  the  OT&E  results  would  be  used  in  the  production  deci¬ 
sion.  The  production  decision  was  made  prior  to  the  test¬ 
ing  for  both  of  the  high  cost  programs  and  one  of  the 
modification  programs.  The  other  two  modification  programs 
made  the  production  decision  after  the  test  results  but 
listed  the  operational  need  and  DT&E  results  as  the  primary 
factors  in  the  decision. 

To  summarize,  the  findings  on  this  research  ques¬ 
tion  revealed  that  test  results  were  being  used  effectively 
by  the  program  decision  makers,  but  not  specifically  as  a 
prime  consideration  in  the  production  decisions.  OT&E 
results  had  the  most  impact  on  the  production  decision 
when  evaluating  off-the-shelf  equipment  in  a  competitive 
test  to  see  which  of  several  systems  would  best  meet  the 
need.  For  non-major  programs  which  involved  essentially 
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known  technology,  OT&E  results  were  not  deemed  as  critical 
to  the  production  decision  as  they  were  to  ensuring  the 
system  was  operationally  ready  for  deployment  in  the  field. 
The  next  chapter  will  discuss  the  major  conclusions  reached 
from  these  findings  and  offer  some  possible  recommenda¬ 
tions  . 
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From  the  review  of  the  literature  it  is  apparent 
that  testing  accomplished  at  the  major  command  level  is 
very  different  from  the  testing  performed  by  AFOTEC.  This 
is  due  to  the  nature  of  the  systems  each  are  tasked  to 
evaluate.  AFOTEC  normally  tests  major  new  acquisition 
systems  that  involve  a  significant  expenditure  of  research 
and  development  funds,  while  the  major  command  GT&E  organi 
nations  normally  test  non-major  system  modifications  or 
off-the-shelf  technology  to  be  adapted  for  military  use. 
The  guidance  provided  for  major  programs  is  much  more 
detailed  and  structured  than  that  provided  for  non-major 
programs.  The  primary  test  and  evaluation  regulations 
offer  very  little  guidance  for  non-major  systems. 

Initial  and/or  qualification  operational  testing 
is  most  often  performed  prior  to  the  production  decision 
at  the  major  command  level  when  there  are  several  systems 
that  are  to  be  evaluated  competitively.  In  these  cases 
the  findings  indicate  that  the  operational  test  results 
are  a  major  factor  in  the  decision  to  proceed  with  produc¬ 
tion.  In  other  cases,  because  the  systems  tested  at  the 
major  command  level  deal  with  essentially  known  technology 
with  known  capabilities,  there  is  not  as  strong  an 


emphasis  on  operational  testing  prior  to  the  production 
decision. 


Even  though  operational  test  results  are  not  always 
a  primary  consideration  in  production  decisions,  they  are 
still  necessary  to  the  decision-making  process.  Test 
results  reveal  to  decision  makers  the  status,  capabilities, 
and  limitations  of  the  system  and  its  readiness  to  be 
fielded.  When  test  results  identify  deficiencies  they 
indicate  areas  that  need  correction  before  the  system  will 
meet  the  stated  need  at  the  required  performance  levels. 
Actions  taken  to  correct  these  deficiencies  help  to  insure 
that  the  systems  eventually  fielded  are  operationally 
effective  and  operationally  suitable. 

The  findings  indicate  that  the  operational  test 
reports  are  sufficient  and  useful  for  providing  valid  input 
into  the  decision-making  process.  The  results  may  not 
always  have  a  major  impact  on  whether  a  system  is  pro¬ 
cured  or  not,  but  they  tend  to  insure  that  the  system 
fielded  performs  as  it  was  intended. 

Recommendations 

The  findings  of  this  research  indicated  several 
areas  in  which  OT&E  might  be  improved.  From  reviewing  the 
regulatory  guidance,  there  appears  to  be  a  lack  of  defini¬ 
tive  guidance  on  the  purpose  and  conduct  of  OT&E  for 
non-major  programs.  The  major  test  and  evaluation 
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regulations  offer  very  little  guidance.  Although  AFR 
55-43  offers  some  guidance  for  conducting  OT&E  for  non¬ 
major  programs,  it  needs  to  be  expanded  at  the  major  com¬ 
mand  level.  The  current  guidance  provided  by  AFCC  is  not 
complete  and  is  somewhat  confusing.  The  AFCC  supplement 
to  AFR  80-14  should  be  expanded  or  a  separate  AFCC  regula¬ 
tion  written  to  provide  clear  and  explicit  guidance  for 
OT&E  conducted  by  AFCC. 

Earlier  involvement  by  the  OT&E  command  with  the 
developing  command  is  recommended.  In  a  few  of  the 
programs  studied  the  1815th  TES  did  not  become  involved 
until  just  prior  to  deployment  of  the  system.  Better  com¬ 
munication  between  AFSC  and  AFCC  could  have  allowed  earlier 
involvement.  This  may  have  led  to  earlier  testing  in  the 
acquisition  process  and  the  scheduling  of  longer  test 
periods  for  more  thorough  testing  of  reliability  and  main¬ 
tainability  specifications.  AFCC  and  AFSC  should  try  to 
insure  that  the  responsible  test  organization  is  brought 
into  a  program  as  early  as  possible. 

Another  area  that  can  be  improved  concerns  the 
readiness  of  systems  to  be  tested.  Logistics  support 
requirements  and  development  testing  which  is  the  responsi¬ 
bility  of  the  contractor  or  the  developing  command  should 
be  completed  before  the  OT&E  test  team  starts  its  tests. 
Better  planning  at  the  program  management  level  could 
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preclude  premature  operational  testing  of  systems  that  are 
not  ready  to  be  tested. 


Areas  for  Further  Research 

In  the  course  of  this  research  there  were  several 
areas  discovered  that  would  benefit  from  further  research. 
One  of  the  areas  is  the  deficiency  reporting  and  tracking 
system  used  by  program  managers  in  AFSC.  A  common  com¬ 
plaint  was  that  deficiencies  are  not  followed  up  in  a 
timely  and  efficient  manner.  Research  in  this  area  would 
be  directed  toward  evaluating  the  present  management  infor¬ 
mation  system  and  comparing  it  with  alternative  methods  of 
deficiency  tracking.  Other  research  in  this  area  could 
examine  the  effectiveness  of  using  a  tracking  system  which 
links  AFSC  to  the  test  organizations  which  identify  the 
deficiencies  and  to  the  supporting  command  which  will  have 
responsibility  for  the  deployed  system. 

Another  area  that  could  benefit  from  further 
research  is  the  evaluation  of  the  effectiveness  of  simula¬ 
tion  techniques  in  OT&E.  The  expense  of  conducting  opera¬ 
tional  tests  can  often  be  prohibitive,  especially  in  the 
testing  of  major  weapon  systems  by  AFOTEC.  Other  tests, 
primarily  concerned  with  survivability,  are  impractical . 

A  research  report  written  by  Lt  Col  Greg  Mann  entitled 
The  Role  of  Simulation  in  Operational  Test  and  Evaluation 
would  be  a  good  starting  point  for  such  research  (20) . 


An  additional  area  for  research  would  be  the  exten¬ 


sion  of  the  research  in  this  thesis  to  the  other  major 
commands,  such  as  Strategic  Air  Command  (SAC)  and  Tactical 
Air  Command  (TAC) ,  to  determine  if  OT&E  conducted  by  these 
commands  has  similar  characteristics  and  limitations  as 
those  discovered  here. 
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Appendix:  List  of  Interviews 


Baughn,  Maj  Lawrence  E.,  Jr.  Program  manager  for  the  Jam 
Resistant  Secure  Communications  Terminal.  Telephone 
interview.  Electronics  Systems  Division  (AFSC) , 
Hanscom  AFB  MA,  25  July  1985. 

Beltran,  Lt  Roger.  Program  manager  for  the  VORTAC  Class 
IV  B  Modification  (AN/FRN-43) .  Telephone  interview. 
Sacramento  Air  Logistics  Center  (AFLC) ,  McClellan  AFB 
CA,  23  July  1985. 

Biram,  Capt  Cliff  M.  Program  manager  for  the  Automated 
Audio  Remote  Test  System  and  the  Digital  Access  and 
Cross  Connect  System.  Telephone  interviews.  HQ  Air 
Force  Communications  Command,  Scott  AFB  IL,  23  July 
and  5  August  1985. 

Boal,  Maj  Robert  H.  III.  Program  monitor  for  the  Ground 
Launched  Cruise  Missile  Convoy /Security  Radios. 
Telephone  interview.  HQ  USAF,  Washington  DC,  30  July 
1985. 

Bradshaw,  TSgt  Gerald  W.  Test  director  for  the  Initial 

Operational  Test  and  Evaluation  of  the  AN/GPN-22  (V) 
Precision  Approach  Radar  ECCM  Improvement.  Personal 
interview.  1815th  Test  and  Evaluation  Squadron 
(AFCC ) ,  Wright-Patter son  AFB  OH,  31  August  1985. 

Gallant,  CMSgt  John  I.  Program  manager  for  the  AN/TPN-19 
Communications  Improvement  Program.  Telephone  inter¬ 
view.  Electronic  Systems  Division  (AFSC) ,  Hanscom 
AFB  MA,  6  August  1985. 

Grant,  SMSgt  Griffin  M.  Test  director  for  the  Qualifica¬ 
tion  Operational  Test  and  Evaluation  of  Scope  Dial. 
Telephone  interview.  2046th  Communications  Group 
(AFCC) ,  Wright-Patterson  AFB  OH,  2  August  1985. 

Green,  Lt  Donald  E. ,  Jr.  Program  manager  for  the  AN/GPN-22 
(V)  Precision  Approach  Radar  ECCM  Improvement. 
Telephone  interview.  Electronic  Systems  Division 
(AFSC),  Hanscom  AFB  MA,  24  July  1985. 
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Harvie,  Lt  Richard  C.  Test  director  for  the  Initial  Opera¬ 
tional  Capability  Combined  Test  Activity  Evaluation 
of  the  Jam  Resistant  Secure  Communications  Terminal. 
Personal  interview.  1815th  Tes  and  Evaluation 
Squadron  (AFCC) ,  Wrignt-Patterson  AFB  OH,  31  July 
1985. 

Heard,  John.  Assistant  program  manager  for  Scope  Signal 
III.  Telephone  interview.  Engineering  Installation 
Division  (AFCC) ,  Tinker  AFB  OH,  2  and  5  August  1985. 

Hughes,  TSgt  Richard  C.  ’'ssociate  test  director  for  the 
Qualification  Operational  Test  and  Evaluation  of  the 
Automated  Audio  Remote  Test  System.  Personal  inter¬ 
view.  1815th  Test  and  Evaluation  Squadron  (AFCC) , 
Wright-Patterson  AFB  OH,  2  August  1985. 

Jessen,  Lt  Karlin  B.  Program  manager  for  the  Transmission 
Monitoring  and  Control  System.  Telephone  interview. 
Electronic  Systems  Division  (AFSC) ,  Hanscom  AFB  MA, 

1  August  1985. 

Johns,  CMSgt  Larry  L.  Test  director  for  the  Qualification 
Operational  Test  and  Evaluation  of  the  VORTAC  Class  IV 
B  Modification  (AN/FRN-43) .  Personal  interview. 

1815th  Test  and  Evaluation  Squadron  AFCC) ,  Wright- 
Patterson  AFB  OH,  31  July  1985. 

Loftin,  Gene.  Associate  test  director  for  the  Special 

Evaluation  of  the  Aeronautical  Systems  Division  Tie 
Line  Network.  Telephone  interview.  Aeronautical 
Systems  Division  (AFSC),  Wright-Patterson  AFB  OH, 

2  August  1985. 

Martin,  Maj  Miles  O.  Project  manager  for  Scope  Dial. 

Telephone  interview.  HQ  Air  Force  Communications 
Command,  Scott  AFB  IL,  30  July  1985. 

McDonald,  Ann  M.  Program  manager  for  the  Fixed  Record 

Communication  Teletypewriter.  Telephone  interview. 
Sacramento  Air  Logistics  Center  (AFLC) ,  McClellan 
AFB  CA,  24  July  1985. 

Musard,  MSgt  Henry  A.  Test  director  for  the  Qualification 
Operational  Test  and  Evaluations  of  Scope  Signal  III 
and  the  Digital  Access  and  Cross  Connect  System. 
Personal  interview.  1815th  Test  and  Evaluation 
Squadron  (AFCC) ,  Wright-Patterson  AFB  OH,  1  August 
1985. 
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Petryk,  MSgu  Charles  W.  Test  director  for  the  Qualifica¬ 
tion  Operational  Test  and  Evaluation  of  the  Ground 
Radio  Interface  Device  (HYX-58).  Personal  interview. 
1815th  Test  and  Evaluation  Squadron  (AFCC) ,  Wright- 
Patterson  AFB  OH,  31  July  1985. 

Poncherri,  Dick.  Dayton  Defense  Metropolitan  Area  Tele¬ 
phone  System  manager.  Telephone  interview.  Aero¬ 
nautical  Systems  Division  (AFSC) ,  Wright-Patterson 
AFB  OH,  2  August  1985. 

Qualters,  MSgt  John  W.  Test  director  for  the  Initial 
Operational  Test  and  Evaluation  of  the  AN/TPN-19 
Communications  Improvement  Program.  Personal  inter¬ 
view.  1815th  Test  and  Evaluation  Squadron  (AFCC) , 
Wright-Patterson  AFB  OH,  1  August  1985. 

Roberts,  Capt  Charles  E.  Test  director  for  the  Qualifica¬ 
tion  Operational  Test  and  Evaluation  of  the  Transmis¬ 
sion  Monitoring  and  Control  System  and  associate  test 
director  for  the  Qualification  Operational  Test  and 
Evaluation  of  the  Ground  Launched  Cruise  Missile 
Convoy /Security  Radios.  Personal  interview.  School 
of  Systems  and  Logistics,  Air  Force  Institute  of 
Technology  (AU) ,  Wright-Patterson  AFB  OH,  1  August 
1985. 

Rodriguez,  Feliciano.  Program  manager  for  the  Ground  Radio 
Interface  Device.  Telephone  interview.  HQ  Air  Force 
Cryptologic  Support  Center  (ESC) ,  San  Antonio  TX, 

1  August  1985. 

Schriml,  Capt  John  M.  Supervisor  of  test  directors  that 

conducted  the  Qualif ication  Test  and  Evaluation  of  the 
Fixed  Record  Communication  Teletypewriter.  Telephone 
interview.  HQ  Air  Force  Communications  Command,  Scott 
AFB  IL,  2  August  1985. 

Schwieterman,  Louis.  Program  manager  for  the  Chemical  War¬ 
fare  Defense  Ensemble  Gloves.  Telephone  interview. 
Aeronautical  Systems  Division  (AFSC) ,  Wright-Patterson 
AFB  OH,  30  July  1985. 
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The  use  of  operational  test  and  evaluation  (OT&E) 
results  in  the  acquisition  decision-making  process  has  been 
increasingly  emphasized  in  the  past  years  and  has  gained 
even  more  prominence  with  the  establishment  of  the  Opera¬ 
tional  Test  and  Evaluation  Office  at  the  Department  of 
Defense.  Special  emphasis  has  been  placed  on  the  use  of 
OT&E  in  the  production  decision.  Initial  Operational  Test 
and  Evaluation  (IOT&E)  and  Qualification  Operational  Test 
and  Evaluation  (QOT&E)  are  the  types  of  test  and  evaluation 
that  are  normally  conducted  on  systems  prior  to  the  first 
’  major  production  decision  in  the  acquisition  process.  While 

IOT&E  usually  evaluates  new  systems  and  QOT&E  usually  evalu¬ 
ates  off-the-shelf  systems,  they  both  provide  program  decision 
makers  with  valid  estimates  of  operational  effectiveness  and 
operational  suitability  upon  which  they  can  base  their  deci¬ 
sions.  This  reiea^h  has  looked  at  how  IOT&E  and  QOT&E 
reports  written  by  the  1815th  Test  and  Evaluation  Squadron 
( AFCC )  are  used  in  the  acquisition  of  non-major  communication 
systems.  The  results  of  this  research  reveal  that  the  QT&E 
conducted,  by  the._1815th  is  usually  QOT&E  performed  on  o'ff-the- 
5  snexf  systems.  Since  these  systems  involve  very  little  tech- 

j  nical  risk  the  OT&E  results  are  not  usually  the  primary  factor 

*  in  production  decisions.  OT&E  results  are  most  useful  in 

•  pointing  out  deficiencies  requiring  correction  prior  to  field- 

j  my  such  systems. 
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